Storage medium having image recognition program stored therein, image recognition apparatus, image recognition system, and image recognition method

ABSTRACT

A game apparatus detects a predetermined image object including a first graphic pattern with a plurality of inner graphic patterns drawn therein from a captured image captured by an image-capturing section. The game apparatus first obtains the captured image captured by the image-capturing section, and detects an area of the first graphic pattern from the captured image. Then, the game apparatus detects the plurality of inner graphic patterns from within the detected area, and calculates center positions of the inner graphic patterns so as to detect the position of the predetermined image object.

CROSS REFERENCE TO RELATED APPLICATION

The disclosure of Japanese Patent Application No. 2010-133510, filed onJun. 11, 2010 and Japanese Patent Application No. 2010-140354, filed onJun. 21, 2010 is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a storage medium having stored thereinan image recognition program for recognizing a predetermined imageobject from an image captured by an image-capturing section, an imagerecognition apparatus, an image recognition system, and an imagerecognition method.

2. Description of the Background Art

There are conventional techniques for detecting a predetermined imageobject from an image (captured image) captured by an image-capturingsection such as a camera. For example, Non-Patent Document 1 (HirokazuKato, Mark Billinghurst, Koichi Asano, Keihachiro Tachibana, “AnAugmented Reality System and its Calibration based on Marker Tracking”,Journal of Virtual Reality Society of Japan, Vol. 4, No. 4, 1999)describes performing an image recognition process on a rectangularmarker included in an image captured by a camera in an augmented realitytechnique. In Non-Patent Document 1, connected regions are extracted bybinarizing the captured image using a fixed threshold value, and theoutline of a connected region is extracted after excluding those of theextracted connected regions that are different from the predeterminedimage object (being too larger or too small, for example). Then, theprocess obtains four line segments from the outline and calculates thevertices of the rectangular marker so as to detect the position of themarker.

With the recognition method described in Non-Patent Document 1, theprocess detects the position (four vertices) of the image object fromthe outline of the image object (marker), and therefore the position ofthe image object may not be detected accurately when the captured imageis blurred due to, for example, a camera shake, or the like. That is, ifthe captured image is blurred, it is possible that the outline of theimage object has not been obtained accurately, and therefore it is notpossible to obtain the accurate position by calculating the position ofthe image object from the outline (see FIG. 7).

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to provide a storagemedium having stored therein an image recognition program capable ofaccurately detecting the position of the image object even if thecaptured image is blurred, an image recognition apparatus, an imagerecognition system, and an image recognition method.

The present invention employs configurations (1) to (16) below to solvethe problems mentioned above.

(1)

The present invention is directed to a computer-readable storage mediumhaving stored therein an image recognition program to be executed by acomputer of an information processing apparatus for detecting apredetermined image object including a plurality of graphic patternsfrom a captured image captured by an image-capturing section. The imagerecognition program instructs the computer to function as: an imageobtaining section, a graphic pattern detection section, and an objectdetection section. The image obtaining section obtains the capturedimage captured by the image-capturing section. The graphic patterndetection section detects the plurality of graphic patterns from thecaptured image. The object detection section detects a position ordirection of the predetermined image object by calculating centerpositions of the plurality of graphic patterns detected by the graphicpattern detection section.

The “image-capturing section” may be built in the information processingapparatus, or may be a separate unit from the information processingapparatus. Where the image-capturing section is built in the informationprocessing section, the image-capturing section may be fixedly attachedto, or detachable from, the information processing apparatus. Theimage-capturing section may operate in cooperation with, orindependently of, the operation of the information processing apparatus.

While the “predetermined image object” is typically a marker, it may beany object as long as it includes a plurality of graphic patterns. Whilethe “graphic pattern” is a plurality of circular graphic patterns in theembodiment to be described later (FIG. 6), the shape, the number, thesize and the color thereof are not limited thereto. For example, the“graphic patterns” may be circular, elliptic or polygonal.

While the “graphic patterns” are inner graphic patterns drawn inside thefirst graphic pattern in the embodiment to be described later, they donot have to be drawn inside a graphic pattern. That is, the “graphicpattern detection section” may detect a plurality of graphic patternsfrom within the area of the first graphic pattern (after detecting thearea of the first graphic pattern) as in the embodiment to be describedlater or the configuration (2) below, or may detect a plurality ofgraphic patterns from the entire area of the captured image (withoutdetecting the area of the first graphic pattern).

The “information processing apparatus” is a concept encompassing anycomputers that process information by executing computer programs, aswell as the game apparatus described in the embodiment to be describedlater. The “information processing apparatus” may or may not be of aportable type.

While the “image recognition program” is, for example, the game programdescribed in the embodiment to be described later, it is a conceptencompassing application programs to be executed by personal computersor portable terminals.

The “image obtaining section” may obtain a captured image from animage-capturing section provided in the information processing apparatusvia an internal bus, or the like, may obtain a captured image from animage-capturing section provided outside the information processingapparatus via telecommunications, or may obtain a captured image from astorage medium storing therein a captured image captured by animage-capturing section when the storage medium is attached to theinformation processing apparatus.

The “graphic pattern detection section” may be any section as long as itdetects the plurality of graphic patterns, and the specific method fordetecting the plurality of graphic patterns may be any method. The“graphic pattern detection section” may detect the area of the graphicpattern as in the embodiment to be described later or the configuration(3) below, for example, may extract the outline of the graphic patternas in the configuration (4) below, or may detect a graphic pattern byusing a pattern matching method.

The “center position of (each) graphic pattern” does not have to be theposition of the center of the graphic pattern in the strict sense, butmay be a position in the vicinity of the center of the graphic patternwhich can be calculated by any of various arithmetic operations. Forexample, the “center position of a graphic pattern” may be calculated asan average value among pixels representing the outline of the graphicpattern, or where pixels of the captured image are weighted in somemanner (with luminance values, etc.), it may be calculated as the centerof gravity among pixels within the area of the graphic pattern.

To “detect a position of a predetermined image object by calculatingcenter positions” only requires that the position of the predeterminedimage object can be identified based on the center positions. Therefore,for example, the “object detection section” may calculate a straightline (approximate straight line) from center positions and identify theposition of the predetermined image object based on the approximatestraight line as in the embodiment to be described later, theconfiguration (6) below or the configuration (16) below, or may use thecenter positions directly as the position of the predetermined imageobject as described in (Variation Regarding Calculation Of MarkerPosition) below. Since the position or direction of the predeterminedimage object within the captured image can be identified from the centerpositions of the plurality of graphic patterns, it is only required thatthe object detection section is capable of detecting at least one of theposition and the direction.

With the configuration (1) above, the position of the predeterminedimage object is detected based on the center positions of the graphicpatterns, not on the outline of the first graphic pattern. Since it isdifficult to accurately detect the outline of the predetermined imageobject if the captured image is blurred, it is not possible toaccurately detect the position of the predetermined image object fromthe outline. In contrast, the center position can be calculatedaccurately even if the captured image is blurred. Therefore, with theconfiguration (1) above, it is possible to accurately calculate theposition or direction of the predetermined image object even if thecaptured image is blurred.

(2)

The predetermined image object may include a first graphic pattern withthe plurality of graphic patterns drawn therein. Then, the imagerecognition program instructs the computer to further function as anarea detection section for detecting an area of the first graphicpattern from the captured image; and. The graphic pattern detectionsection detects the plurality of graphic patterns from within the areadetected by the area detection section.

The “area detection section” may be any section as long as it detectsthe first graphic pattern of the predetermined image object, and thespecific method for detecting the first graphic pattern may be anymethod. For example, the “area detection section” may detect the firstgraphic pattern by extracting the outline of the first graphic pattern,or may detect the first graphic pattern by using a pattern matchingmethod.

With the configuration (2) above, first, the area of the first graphicpattern is detected, and the graphic patterns (inner graphic patterns)drawn inside the first graphic pattern are detected from within thedetected area. Then, it is possible to prevent an erroneous detection ofdetecting an object that is not the predetermined image object as beingthe inner graphic pattern, and it is possible to reliably detect theinner graphic patterns.

(3)

The graphic pattern detection section may detect areas of the pluralityof graphic patterns. Then, the object detection section includes aposition calculation section for calculating center positions of theareas detected by the graphic pattern detection section.

With the configuration (3) above, center positions can be easilycalculated by detecting the areas of the plurality of graphic patternsand then calculating the center positions of the detected areas.

(4)

The graphic pattern detection section may detect outlines representingthe areas of the plurality of graphic patterns. Then, the positioncalculation section calculates center positions of the areas representedby the outlines detected by the graphic pattern detection section.

With the configuration (4) above, center positions can be easilycalculated by detecting the outlines of the areas of the plurality ofgraphic patterns and then calculating the center positions of thedetected outlines.

(5)

The position calculation section may calculate the center position ofeach of the areas by obtaining positions of pixels representing theoutline detected by the graphic pattern detection section andcalculating an average value among the obtained positions.

With the configuration (5) above, the center position can be calculatedeasily and accurately by calculating the average value among thepositions of pixels representing the outline.

(6)

The object detection section may include a straight line calculationsection for calculating a predetermined straight line based on thecenter positions of the plurality of graphic patterns. Then, the imagerecognition program instructs the computer to further function as apositional relationship calculation section for calculating a positionalrelationship between the predetermined image object and theimage-capturing section based on the predetermined straight linecalculated by the straight line calculation section.

With the configuration (6) above, the position and direction of thepredetermined image object in the captured image can be identified bythe straight line calculated based on the center positions. Therefore,the positional relationship can be calculated accurately based on thestraight line.

With the configuration (6) above, the virtual image of the embodiment tobe described later can be synthesized with the captured image by usingthe positional relationship. That is, with the configuration (6) above,in an augmented reality technique for displaying a synthesized imageobtained by synthesizing together the captured image and the virtualimage, it is possible to use a recognition process that is resistant tothe blur of the captured image (capable of accurately detecting thepredetermined image object even if the captured image is blurred). Notethat in an augmented reality technique, it is assumed that it is oftenthe case where the user uses the image-capturing section while moving itaround, and it is believed that that is particularly often the case whenthe augmented reality technique is used in a game as in the embodimentto be described later. Therefore, where the configuration (6) above isused in an augmented reality technique, the recognition process of theconfiguration (1) above which is resistant to the blur of the image isparticularly effective.

(7)

Each of the plurality of graphic patterns may be circular.

With the configuration (7) above, the shape of the graphic pattern inthe captured image remains the same irrespective of the direction inwhich the predetermined image object lies. Therefore, the centerposition does not vary depending on the direction of the predeterminedimage object, and it is therefore possible to more accurately detect thepredetermined image object.

(8)

The area detection section may detect an area of the first graphicpattern based on pixel values obtained at a first pitch across thecaptured image. Then, the object detection section calculates centerpositions of the graphic patterns based on pixel values obtained at asecond pitch smaller than the first pitch across the area of thecaptured image detected by the area detection section.

The “second pitch” may be of any length as long as it is smaller thanthe “first pitch”. Where a plurality of different processes areperformed in the image object detecting process, as in the embodiment tobe described later, the object detection section may perform the processof calculating the center position based on pixel values obtained at thesecond pitch, while the other processes may be performed with anyprecision.

With the configuration (8) above, after first detecting the firstgraphic pattern with a low precision by the area detection section, theprocess of calculating the center positions of the plurality of graphicpatterns within the first graphic pattern is performed with a highprecision. Thus, the process of detecting the first graphic pattern isperformed with a low precision, and the range across which the processof detecting the plurality of graphic patterns is performed is limitedto within the first graphic pattern, and it is therefore possible toperform the recognition process in a short period of time. Since theprocess of calculating the center positions of the detected graphicpatterns is performed with a high precision, it is possible to detectthe position of the predetermined image object precisely. Therefore,with the configuration (8) above, it is possible to provide arecognition process with which it is possible to shorten the amount oftime required for detection while maintaining the object detectionprecision.

(9)

The image recognition program may instruct the computer to furtherfunction as an outline extraction section for extracting an outline ofthe first graphic pattern in the area of the first graphic patterndetected by the area detection section. Then, the object detectionsection detects the plurality of graphic patterns from an area withinthe outline extracted by the outline extraction section.

With the configuration (9) above, the graphic patterns are detected fromwithin the area of the outline extracted by the outline extractionsection, and it is therefore possible to appropriately determine therange across which the detection of the graphic patterns is performed.Thus, it is possible to prevent the graphic pattern detection processfrom being performed wastefully on unnecessary areas, and it istherefore possible to more efficiently perform the recognition process.

(10)

The outline extraction section may extract the outline of the firstgraphic pattern based on pixel values obtained at a predetermined pitchacross the captured image. Then, the object detection section detectsthe plurality of graphic patterns from the area within the outline basedon pixel values obtained at a pitch smaller than the predetermined pitchacross the captured image.

With the configuration (10) above, the outline extracting process isperformed with a lower precision than the graphic pattern detectionprocess, and it is therefore possible to perform the outline extractingprocess in a shorter period of time. As a result, it is possible toperform the recognition process in a shorter period of time.

(11)

The image recognition program may instruct the computer to function as ashape determination section for determining whether the outlineextracted by the outline extraction section represents a shape of thefirst graphic pattern. Then, the object detection section detects theplurality of graphic patterns from within an area corresponding to theoutline which is determined by the shape determination section torepresent a shape of the first graphic pattern.

With the configuration (11) above, it is possible to determine, based onthe extracted outline, whether the first graphic pattern is includedwithin the area detected by the area detection section. Then, where thefirst graphic pattern is not included within the area, the graphicpattern detection process is not performed for that area. Therefore,with the configuration (11) above, areas that do not include the firstgraphic pattern can be excluded from the graphic pattern detectionprocess, and it is therefore possible to prevent the graphic patterndetection process from being performed wastefully. Thus, it is possibleto efficiently perform the recognition process and to perform therecognition process in a shorter period of time. Particularly, where theconfigurations (10) and (11) above are combined with each other, thegraphic pattern detection process which requires a relatively long time(as it is performed with a high precision) can be omitted by virtue ofthe outline extracting process which is performed relatively quickly (asit is performed with a low precision), and it is therefore possible toperform the recognition process more efficiently.

(12)

The predetermined image object may include a second graphic pattern at aposition different from the first graphic pattern. Then, the imagerecognition program instructs the computer to further function as adirection identification section for detecting the second graphicpattern from the captured image and identifying a direction of thepredetermined image object based on a direction in which the secondgraphic pattern is arranged with respect to the first graphic pattern.

With the configuration (12) above, the direction of the predeterminedimage object can be identified based on the direction of the secondgraphic pattern with respect to the first graphic pattern. With theconfiguration (12) above, since the second graphic pattern is formed ata position of the predetermined image object different from the firstgraphic pattern, it is possible to have only the inside graphic patternsdrawn inside the first graphic pattern. Therefore, when detecting theinside graphic patterns, it is possible to accurately detect the insidegraphic patterns without erroneously detecting the second graphicpattern or a part thereof. Therefore, it is possible to further improvethe precision of the recognition process.

(13)

The graphic pattern detection section may detect the areas of theplurality of graphic patterns based on pixel values obtained at apredetermined pitch across the captured image. Then, the positioncalculation section calculates center positions of the areas detected bythe graphic pattern detection section based on pixel values obtained ata pitch smaller than the predetermined pitch across the captured image.

With the configuration (13) above, since the process of detecting theareas of the plurality of graphic patterns is performed with a lowerprecision than the center position calculation process, it is possibleto perform the process of detecting the areas of the plurality ofgraphic patterns in a shorter period of time. As a result, it ispossible to perform the recognition process in a shorter period of time.

(14)

The object detection section may further include a number-of-areasdetermination section for determining whether the number of areasdetected by the graphic pattern detection section is a number within apredetermined range. Then, the position calculation section calculatescenter positions of the areas detected by the graphic pattern detectionsection for the area of the first graphic pattern where a number ofareas within the predetermined range have been detected.

With the configuration (14) above, it is possible to determine whetherthe number of areas which have been detected by the graphic patterndetection section is an appropriate number. If the number of areas isnot appropriate, it is determined that the first graphic pattern is notincluded in the area detected by the area detection section, and theprocess of calculating the center positions is not performed. Therefore,with the configuration (14) above, the areas that do not include thefirst graphic pattern can be excluded from the center positioncalculation process, and it is therefore possible to prevent thecalculation process from being performed wastefully. Thus, it ispossible to efficiently perform the recognition process and to performthe recognition process in a shorter period of time. Particularly, wherethe configurations (13) and (14) above are combined with each other, theprocess of calculating the center positions which requires a relativelylong time (as it is performed with a high precision) can be omitted byvirtue of the process of detecting the area of the first graphic patternwhich is performed relatively quickly (as it is performed with a lowprecision), and it is therefore possible to perform the recognitionprocess more efficiently.

(15)

The predetermined image object may include three or more graphicpatterns. Then, the object detection section detects a position anddirection of the predetermined image object based on center positions ofthe three or more graphic patterns.

To “detect the direction (of the predetermined image object)” is notlimited to uniquely identifying the direction of the predetermined imageobject. For example, while the process of step S26 calculates thepositions of the four vertices from the center positions in theembodiment to be described later, it is not possible, based on thepositions of these four vertices, to distinguish between the marker 51being rotated by 90°, 180° or 270° and being not rotated. The objectdetection section in the configuration (15) above may be a process thatdoes not uniquely identify the direction of the predetermined imageobject, as with the process of step S26 described above.

With the configuration (15) above, it is possible to identify thepositions (in the captured image) of three points in the predeterminedimage object by calculating the center positions of the three or moregraphic patterns, and it is therefore possible to detect the positionand direction of the predetermined image object.

(16)

A predetermined number, three or more, of graphic patterns may bearranged on a predetermined straight line in the predetermined imageobject. Then, the object detection section calculates the predeterminedstraight line based at least on a number of center positions among allcenter positions calculated, the number being two or more and smallerthan the predetermined number, so as to detect a position and directionof the predetermined image object based on the calculated predeterminedstraight line.

With the configuration (16) above, it is possible to detect the positionand direction of the predetermined image object based on the positionand direction of the predetermined straight line in the captured image.Herein, the predetermined straight line is calculated from the centerpositions of a number of graphic patterns which is less than apredetermined number, and it is therefore possible to calculate thepredetermined straight line even if a predetermined number of graphicpatterns are not all detected. Therefore, with the configuration (16)above, there is provided redundancy in the number of graphic patterns,whereby it is possible to detect the position and direction of thepredetermined image object even if some of the graphic patterns are notdetected. Therefore, it is possible to more reliably detect thepredetermined image object, and thus to increase the success rate of therecognition process.

The present invention may also be embodied in the form of an imagerecognition apparatus including various sections similar to thosedescribed above. In such an image recognition apparatus, the varioussections may be implemented by a computer executing an image recognitionprogram, or some or all of the various sections may be implemented by adedicated circuit or circuits. The present invention may also beembodied in the form of an image recognition system including one ormore information processing apparatuses each having various sectionsdescribed above. Then, the one or more information processingapparatuses may communicate with each other directly via a wired orwireless connection, or may communicate with each other via a network.Moreover, the present invention may also be embodied in the form of animage recognition method performed by the various sections describedabove.

The present invention may also be embodied in the form of a marker usedin an augmented reality technique. That is, the marker is used in animage display system, in which a captured image captured by animage-capturing section is subjected to a predetermined recognitionprocess to calculate a position and direction of the marker within thecaptured image, and an image of a virtual object is produced based onthe position and direction of the marker, so as to display an imageobtained by synthesizing the image of the virtual object with thecaptured image. The marker includes a plurality of inner graphicpatterns. The center positions of the plurality of inner graphicpatterns are calculated (by the image display system). The centerpositions are used for calculating (by the image display system)information representing the position and direction of the marker. Themarker may be a thin plate-like member with a graphic pattern drawnthereon as in the embodiment to be described later, or may beimplemented by a graphic pattern displayed on the screen of a displayapparatus.

According to the present invention, the process calculates the centerpositions of a plurality of graphic patterns included in a predeterminedimage object, and detects the position of the predetermined image objectbased on the center positions, and it is therefore possible toaccurately calculate the position of the predetermined image object evenif the captured image is blurred.

These and other objects, features, aspects and advantages of the presentinvention will become more apparent from the following detaileddescription of the present invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external view of a game apparatus of the presentembodiment;

FIG. 2 is a block diagram showing an example of an internalconfiguration of the game apparatus;

FIG. 3 is a view showing an example of how the game apparatus is used;

FIG. 4 is a view showing an example of a game image displayed on thegame apparatus;

FIG. 5 is a diagram showing an object area used in an image recognitionprocess of the present embodiment;

FIG. 6 is a diagram showing a marker used in the present embodiment;

FIG. 7 is a diagram showing an image of a first graphic pattern with theleft and right portions of its outline blurred;

FIG. 8 is a diagram showing the center position of an inner graphicpattern in a part of the first graphic pattern;

FIG. 9 is a diagram showing the center points of the inner graphicpatterns in the first graphic pattern;

FIG. 10 is a diagram showing various data to be used in processesperformed by a game program;

FIG. 11 is a main flow chart showing the flow of a game process executedby the game apparatus;

FIG. 12 is a flow chart showing the flow of a recognition process (stepS2) shown in FIG. 11;

FIG. 13 is a diagram showing pixels in the captured image whose pixelvalues are referenced in the process of step S11;

FIG. 14 is a flow chart showing the flow of a marker detection process(step S15) shown in FIG. 12;

FIG. 15 is a diagram showing an example of an area to be the subject ofthe process of in step S21;

FIG. 16 is a diagram showing pixels in the captured image whose pixelvalues are referenced in the process of step S21;

FIG. 17 is a diagram showing an example of a captured image representinga first graphic pattern;

FIG. 18 is a diagram showing four vertices P11 to P14 calculated fromstraight lines L1 to L4 shown in FIG. 17;

FIG. 19 is a diagram showing an example of a predetermined area set instep S27;

FIG. 20 is a diagram showing a marker of a variation of the presentembodiment; and

FIG. 21 is a diagram showing a marker of another variation of thepresent embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[Hardware Configuration of Game Apparatus]

Referring to the drawings, an image recognition program and an imagerecognition apparatus according to one embodiment of the presentinvention will be described. While the present invention is realized byexecuting an image recognition program by any information processingapparatus (computer) which displays an image on a display apparatus, thepresent embodiment is directed to a case where a game apparatus 1 shownin FIG. 1 is used as an example of the information processing apparatus.

FIG. 1 is an external view of the game apparatus 1 according to thepresent embodiment. Here, a hand-held game apparatus is shown as anexample of the game apparatus 1. Note that the game apparatus 1 includesa camera and functions as an image-capturing device for capturing animage with the camera, displaying the captured image on the screen, andstoring data for the captured image.

In FIG. 1, the game apparatus 1 is a foldable hand-held game apparatus,and the game apparatus 1 shown in the figure is unfolded (open state).The game apparatus 1 is structured in a size allowing the user to holdit with both hands, or even one hand, when unfolded.

The game apparatus 1 has a lower housing 11 and an upper housing 21. Thelower housing 11 and the upper housing 21 are connected so as to beopenable/closable (foldable). In the example of FIG. 1, the lowerhousing 11 and the upper housing 21 are each formed in the shape of ahorizontally elongated rectangular plate, and connected so as to bepivotable at their common long side joint. Typically, the user uses thegame apparatus 1 in open state. Also, when the user does not use thegame apparatus 1, the game apparatus 1 is stored in closed state. Inaddition, in the example shown in FIG. 1, the game apparatus 1 canmaintain not only the closed and open states but also its opening statevia friction force generated at the joint at any angle that can be madeby the lower housing 11 and the upper housing 21 between the closed andopen states. That is, the upper housing 21 can remain stationary at anarbitrary angle with respect to the lower housing 11.

The lower housing 11 is provided with a lower LCD (Liquid CrystalDisplay) 12. The lower LCD 12 is horizontally long, and is arranged suchthat its longitudinal direction coincides with the longitudinaldirection of the lower housing 11. Note that in the present embodiment,LCDs are used as display devices to be included in the game apparatus 1,but any other display devices, such as EL (Electra Luminescence) displaydevices, may be used. In addition, display devices of any resolution canbe used for the game apparatus 1. Note that an image being captured byan internal camera 23 or an external camera 25 is displayed in real-timeon lower LCD 12.

The lower housing 11 is provided with various operation buttons 14A to14K and a touch panel 13 as input devices. As shown in FIG. 1, of allthe operation buttons 14A to 14K, the direction input button 14A, theoperation button 14B, the operation button 14C, the operation button14D, the operation button 14E, the power button 14F, the start button14G, and the select button 14H are provided at the inner principalsurface of the lower housing 11. The inner principal surface is asurface to be the interior side when the upper housing 21 and the lowerhousing 11 are folded. In the example shown in FIG. 1, the directioninput button 14A and the power button 14F are provided to the left orright (in FIG. 1, to the left) of a lower LCD 12 provided at the centerof the inner principal surface of the lower housing 11. In addition, theoperation buttons 14B to 14E, the start button 14G, and the selectbutton 14H are provided to the opposite side (in FIG. 1, to the right)of the lower LCD 12 on the inner principal surface of the lower housing11. The direction input button 14A, the operation buttons 14B to 14E,the start button 14G, and the select button 14H are used for variousoperations on the game apparatus 1. For example, the direction inputbutton 14A is used for selection operations and so on. The operationbuttons 14B to 145 are used for setting and cancellation operations andso on. The power button 14F is used for turning ON/OFF the gameapparatus 1.

Note that in FIG. 1, the operation buttons 14I to 14K are not shown. Forexample, the L button 14I is provided at the left edge of the uppersurface of the lower housing 11, and the R button 14J is provided at theright edge of the upper surface of the lower housing 11. For example,the L button 14I and the R button 14J are used for image-capturinginstruction operations (shutter operations) on the game apparatus 1having an image-capturing function. Furthermore, the volume button 14Kis provided at the left side surface of the lower housing 11. The volumebutton 14K is used for controlling the volume of a speaker included inthe game apparatus 1.

Also, in addition to the operation buttons 14A to 14K, the gameapparatus 1 further includes a touch panel 13 as an example of apointing device that is an input device allowing designation of anarbitrary position on the screen. The touch panel 13 is attached so asto cover the screen of the lower LCD 12. Note that in the presentembodiment, a touch panel 13 is, for example, of a resistive film type.However, a touch panel 13 is not limited to the resistive film type, andmay be of any type. Also, in the present embodiment, for example, thetouch panel 13 has the same resolution (detection accuracy) as the lowerLCD 12. However, the touch panel 13 is not necessarily required to beequal in resolution to the lower LCD 12. In addition, the lower housing11 has an insertion opening (in FIG. 1, indicated by broken lines)provided in its right side surface. The insertion opening can store atouch pen 27 to be used for operating the touch panel 13. Note that anyinput to the touch panel 13 (touch input) is normally performed with thetouch pen 27, but the touch pen 27 is not restrictive, and the touchpanel 13 can be operated with the user's finger.

Also, the lower housing 11 has provided in the right side surface aninsertion opening (in FIG. 1, indicated by two-dot chain lines) forstoring a memory card 28. The insertion opening has provided therein aconnector (not shown) for electrically connecting the game apparatus 1and the memory card 28. The memory card 28 is, for example, an SD(Secure Digital) memory card removably attached to the connector. Forexample, the memory card 28 is used for storing (saving) images capturedby the game apparatus 1 and reading into the game apparatus 1 imagesgenerated by other apparatuses.

Furthermore, the lower housing 11 has provided in its upper side surfacean insertion opening (in FIG. 1, indicated by one-dot chain lines) forstoring a memory card 29. This insertion opening also has providedtherein a connector (not shown) for electrically connecting the gameapparatus 1 and the memory card 29. The memory card 29 is a storagemedium having an information processing program such as a game programstored therein, and is removably loaded into the insertion openingprovided in the lower housing 11.

Three LEDs 15A to 15C are provided to the left of the joint between thelower housing 11 and the upper housing 21. Here, the game apparatus 1 iscapable of wirelessly communicating with other equipment, and the firstLED 15A is lit up while the game apparatus 1 is ON. The second LED 15Bis lit up while the game apparatus 1 is being charged. The third LED 15Cis lit up when wireless communication is established. Accordingly, thethree LEDs 15A to 15C allow the user to know the statuses of the gameapparatus 1, regarding ON/OFF of the power supply, battery charge, andcommunications.

On the other hand, the upper housing 21 is provided with an upper LCD22. The upper LCD 22 is horizontally long, and is arranged such that itslongitudinal direction coincides with the longitudinal direction of theupper housing 12. Note that as in the case of the lower LCD 12, anydisplay device of any other type and resolution may be used instead ofthe upper LCD 22. Note that a touch panel may be provided over the upperLCD 22. The upper LCD 22 displays, for example, an operation instructionscreen for teaching the user about the roles of the operation buttons14A to 14K and the touch panel 13.

Also, the upper housing 21 is provided with two cameras (an internalcamera 23 and an external camera 25). As shown in FIG. 1, an internalcamera 23 is provided at the inner principal surface close to the jointof the upper housing 21. On the other hand, the external camera 25 isprovided opposite to the side of the inner principal surface where theinternal camera 23 is provided, i.e., the external principal surface ofthe upper housing 21 (the exterior surface of the game apparatus 1 inclosed state; the back of the upper housing 21 shown in FIG. 1). Notethat in FIG. 1, the external camera 25 is indicated by a broken circle.As a result, the internal camera 23 can capture images of the directionin which the inner principal surface of the upper housing 21 isoriented, while the external camera 25 can capture images of thedirection opposite to the image-capturing direction of the internalcamera 23, i.e., the direction in which the external principal surfaceof the upper housing 21 is oriented. In this manner, in the presentembodiment, the two cameras, i.e., the internal and external cameras 23and 25, are provided so as to capture images in their respectivedirections opposite to each other. For example, the user can use theinternal camera 23 to capture images of a view from the game apparatus 1toward the user, and also can use the external camera 25 to captureimages of a view in the opposite direction, i.e., from the user towardthe game apparatus 1.

Note that a microphone (microphone 43 shown in FIG. 2) is provided as anaudio input device under the inner principal surface close to the joint.In addition, a microphone hole 16 is provided in the inner principalsurface close to the joint such that the microphone 43 can sense soundfrom outside the game apparatus 1. The microphone 43 and the microphonehole 16 are not necessarily required to be positioned at the joint. Forexample, the microphone 43 may be accommodated within the lower housing11, and the microphone hole 16 may be provided in the lower housing 11at a position corresponding to the position where the microphone 43 isaccommodated.

Also, a fourth LED 26 (in FIG. 1, indicated by a broken circle) isprovided at the external principal surface of the upper housing 21. Thefourth LED 26 is lit up at the time the internal camera 23 or theexternal camera 25 captures an image (when the shutter button ispressed). Furthermore, the light is on while the internal camera 23 orthe external camera 25 is capturing a motion picture. As such, thefourth LED 26 allows any subject and bystander to know the gameapparatus 1 captured (or is capturing) a picture.

Also, a sound hole 24 is provided to both the left and the right of anupper LCD 22 provided at the center of the inner principal surface ofthe upper housing 21. A speaker is accommodated within the upper housing21 below each sound hole 24. The sound hole 24 is a hole for emanatingthe sound from the speaker to the outside of the game apparatus 1.

As described above, the upper housing 21 is provided with the internaland external cameras 23 and 25 which are image-capturing means forcapturing images, as well as the upper LCD 22 acting as a display meansfor mainly displaying the operation instruction screen. On the otherhand, the lower housing 11 is provided with the input devices (the touchpanel 13 and the operation buttons 14A to 14K) for operational inputs tothe game apparatus 1, and the lower LCD 12 acting as a display means fordisplaying captured images. Accordingly, when using the game apparatus1, the user can see a captured image (an image captured by the camera)displayed on the lower LCD 12 and make inputs via the input deviceswhile holding the lower housing 11.

Next, an internal configuration of the game apparatus 1 will bedescribed with reference to FIG. 2. FIG. 2 is a block diagramillustrating an exemplary internal configuration of the game apparatus1.

In FIG. 2, the game apparatus 1 includes electronic parts, such as a CPU31, a main memory 32, a memory control circuit 33, a saved data memory34, a preset data memory 35, a memory card interface (memory card I/F)36, a memory card I/F 37, a wireless communication module 38, a localcommunication module 39, a real time clock (RTC) 40, a power circuit 41,an interface circuit (I/F circuit) 42, a first GPU (Graphics ProcessingUnit) 45, a second GPU 46, a first VRAM (Video RAM) 47, a second VRAM48, and an LCD controller 49. These electronic parts are mounted on anelectronic circuit board, and accommodated within the lower housing 11(or may be accommodated within the upper housing 21).

The CPU 31 is an information processing means for executing apredetermined program (here, an image display program according to thepresent embodiment). In the present embodiment, the game program isstored, as an example of the image display program, in a memory (e.g.,saved data memory 34) within the game apparatus 1 as well as in thememory card 28 and/or 29, and the CPU 31 executes the game program,thereby executing a game process to be described later. Note that theprogram to be executed by the CPU 31 may be prestored in the memorywithin the game apparatus 1 or may be acquired from the memory card 28and/or 29 or from other equipment through communication therewith.

The CPU 31 is connected to the main memory 32, the memory controlcircuit 33, and the preset data memory 35. The memory control circuit 33is connected to the saved data memory 34. The main memory 32 is astorage means used as a working area or buffering area for the CPU 31.Specifically, a main memory 32 stores various data to be used in thegame process, and programs acquired from outside (e.g., the memory cards28 and 29 and other equipment). In the present embodiment, for example,a PSRAM (Pseudo-SRAM) is used as the main memory 32. The saved datamemory 34 is a storage means for storing, for example, the programs tobe executed by the CPU 31 and data for images captured by the internalcamera 23 and the external camera 25. The saved data memory 34 isconfigured by a nonvolatile storage medium, e.g., in the presentembodiment, a NAND flash memory. The memory control circuit 33 is acircuit for controlling data reading from/writing to the saved datamemory 34 in accordance with an instruction by the CPU 31. The presetdata memory 35 is a storage means for storing data (preset data) such asvarious present parameters for the game apparatus 1. As for the presetdata memory 35, a flash memory connected to the CPU 31 via an SPI(Serial Peripheral Interface) bus can be used.

The memory card I/Fs 36 and 37 are each connected to the CPU 31. Thememory card I/F 36 reads/writes data from/to the memory card 28 attachedto the connector in accordance with an instruction from the CPU 31.Also, the memory card I/F 37 reads/writes data from/to the memory card29 attached to the connector in accordance with an instruction from theCPU 31. In the present embodiment, image data captured by the internalcamera 23 and the external camera 25, as well as image data receivedfrom other devices are written into the memory card 28, and image datastored in the memory card 28 is read from the memory card 28 and storedto the saved data memory 34 or transmitted to other devices. Inaddition, various programs stored in the memory card 29 are read andexecuted by the CPU 31.

Note that the game program may be supplied to a computer system not onlyvia an external storage medium, such as the memory card 29, but also viaa wired or wireless communication line. Also, the game program may bepre-recorded to a nonvolatile storage device within the computer system.Note that the information storage medium for storing the game program isnot limited to the nonvolatile storage device, and may be a CD-ROM, aDVD, or a similar optical disk storage medium.

The wireless communication module 38 has a function of connecting to awireless LAN in accordance with a system complying with, for example,the IEEE802.11.b/g standard. Also, the local communication module 39 hasa function of wirelessly communicating with similar game apparatuses inaccordance with a predetermined communication system. The wirelesscommunication module 38 and the local communication module 39 areconnected to the CPU 31. The CPU 31 is capable of transmitting/receivingdata to/from other equipment via the Internet using the wirelesscommunication module 38, as well as transmitting/receiving data to/fromother similar game apparatuses via the Internet using the localcommunication module 39.

The CPU 31 is also connected to the RTC 40 and the power circuit 41. TheRTC 40 counts time and provides an output to the CPU 31. For example,the CPU 31 can calculate the current time (date) based on the timecounted by the RTC 40. The power circuit 41 controls power supplied fromthe power supply (typically, a battery accommodated in the lower housing11) provided in the game apparatus 1, and supplies power to variousparts of the game apparatus 1.

The game apparatus 1 is also provided with the microphone 43 and anamplifier 44. The microphone 43 and the amplifier 44 are each connectedto the I/F circuit 42. The microphone 43 senses the voice of the userspeaking to the game apparatus 1, and outputs an audio signalrepresenting the voice to the I/F circuit 42. The amplifier 44 amplifiesthe audio signal from the I/F circuit 42 to provide an output from thespeaker (not shown). The I/F circuit 42 is connected to the CPU 31.

Also, the touch panel 13 is connected to the I/F circuit 42. The I/Fcircuit 42 includes an audio control circuit for controlling themicrophone 43 and the amplifier 44 (speaker), and a touch panel controlcircuit for controlling the touch panel 13. The audio control circuitperforms A/D conversion and D/A conversion on the audio signal, and alsoconverts the audio signal into audio data of a predetermined format. Thetouch panel control circuit generates touch position data (detectedcoordinate data to be described later) of a predetermined format basedon a signal from the touch panel 13, and outputs the generated data tothe CPU 31. The touch position data is data representing coordinates ofa position detected by the touch panel 13 as being the position at whichan input was made to the input screen of the touch panel 13. Note thatthe touch panel control circuit performs reading of a signal from thetouch panel 13 and generation of detected coordinate data once everypredetermined period of time.

The above-described operation buttons 14A to 14K constitute an operationbutton section 14 connected to the CPU 31. The operation button section14 outputs to the CPU 31 operation data representing the status of inputto the operation buttons 14A to 14K (whether or not the buttons havebeen pressed). The CPU 31 acquires the operation data from the operationbutton section 14, and executes a process in accordance with an input tothe operation button section 14.

The internal camera 23 and the external camera 25 are each connected tothe CPU 31. The internal camera 23 and the external camera 25 eachcapture an image in accordance with an instruction from the CPU 31, andoutput data for the captured image to the CPU 31. In the presentembodiment, the CPU 31 gives an image-capturing instruction to eitherthe internal camera 23 or the external camera 25, and the camerareceiving the image-capturing instruction captures an image andtransmits image data to the CPU 31. Note that the internal camera 23 andthe external camera 25 are also capable of capturing motion pictures.That is, the internal camera 23 and the external camera 25 are alsocapable of repeatedly capturing images and repeatedly sending thecaptured data to the CPU 31.

The first GPU 45 is connected to the first VRAM 47, and the second GPU46 is connected to the second VRAM 48. In accordance with an instructionfrom the CPU 31, the first GPU 45 generates a first display image basedon display image generation data stored in the main memory 32, andcreates an image on the first VRAM 47. In accordance with an instructionfrom the CPU 31, the second GPU 46 generates a second display image, andcreates an image on the second VRAM 48, as in the case of the first GPU45. The first VRAM 47 and the second VRAM 48 are connected to the LCDcontroller 49.

The LCD controller 49 includes a register 491. The register 491 storesthe value of 0 or 1 in accordance with an instruction from the CPU 31.When the value in the register 491 is 0, the LCD controller 49 outputsthe first display image created on the first VRAM 47 to the lower LCD12, and also outputs the second display image created on the second VRAM48 to the upper LCD 22. Alternatively, when the value in the register491 is 1, the LCD controller 49 outputs the first display image createdon the first VRAM 47 to the upper LCD 22, and also outputs the seconddisplay image created on the second VRAM 48 to the lower LCD 12. Forexample, the CPU 31 is capable of causing the lower LCD 12 to display animage acquired from either the internal camera 23 or the external camera25, while causing the upper LCD 22 to display an operation instructionscreen generated by a predetermined process.

[Outline of Game Process]

Next, game processes to be performed during a game process executed bythe game program will be described with reference to FIGS. 3 to 9. Thegame program performs an image recognition process on an image (capturedimage) of the real space captured by a camera, and allows a player toplay a game while displaying a game image obtained by synthesizing animage (virtual image) of the virtual space with the captured image basedon the results of the image recognition process.

FIG. 3 is a view showing an example of how the game apparatus is used.In the present embodiment, the player (user) places a marker 51 as apredetermined image object at any place (on a table 50 in FIG. 3), anduses the game apparatus 1 to capture an image of the marker 51 and thevicinity thereof, as shown in FIG. 3. While the camera used for imagecapturing may be either the internal camera 23 or the external camera25, an example where the external camera 25 is used will be describedherein. In the present embodiment, the marker 51 is a thin plate-likemarker with a predetermined pattern (see FIG. 6) drawn thereon.

FIG. 4 is a view showing an example of a game image displayed on thegame apparatus. FIG. 4 shows a game image to be displayed on the upperLCD 22 of the game apparatus 1 when the game apparatus 1 captures animage of the marker 51 placed on the table 50 as shown in FIG. 3. Notethat while the game image is displayed on the upper LCD 22 in thepresent embodiment, the game image may be displayed on the lower LCD 12.As shown in FIG. 4, an image of a cannon 52, as an image (virtual image)of a virtual object in the virtual space, is displayed on the upper LCD22 while being synthesized with the captured image. Thus, it is possibleto display an image which looks as if the cannon 52 were present on theactual table 50. Note that while FIG. 4 shows an image of the cannon 52as an example of the virtual object, the virtual object may be anyobject. Thus, in the present embodiment, a game image is displayed as ifa virtual object were present in the real space by using an augmentedreality technique.

Note that an image (synthesized image) obtained by synthesizing avirtual image with a captured image can be produced by the followingprocess, for example. That is, the game apparatus 1 first performs an(image) recognition process of recognizing the marker 51 contained inthe image captured by the camera. Here, as process results of therecognition process, information representing the position and directionof the marker 51 in the captured image is calculated. When the marker 51is recognized, the game apparatus 1 calculates the positionalrelationship between the game apparatus 1 (camera) and the marker 51from the position and direction of the marker 51 which are processresults of the recognition process. Moreover, the game apparatus 1calculates the position and orientation of the virtual camera in thevirtual space based on the positional relationship. The position andorientation of the virtual camera are calculated so that the positionalrelationship between the virtual camera and the virtual object in thevirtual space coincides with the positional relationship between thegame apparatus 1 and the marker 51 in the real space. Once the positionof the virtual camera is determined, the game apparatus 1 produces avirtual image showing the virtual object as viewed from the position ofthe virtual camera, and synthesizes the virtual image with the capturedimage. By the process described above, the game apparatus 1 can produceand display a synthesized image. Note that the process of calculatingthe position of the virtual camera from the positional relationship maybe similar to a process of a conventional augmented reality technique.

[Outline of Image Recognition Process]

As described above, in the present embodiment, the game apparatus 1detects the marker 51 from the captured image by the recognitionprocess. If the marker 51 cannot be accurately detected in therecognition process, it is not possible to accurately produce thevirtual image, and it is then no longer possible to synthesize thevirtual image at a correct position on the captured image, detractingfrom the realness of the synthesized image. Therefore, in the presentembodiment, the game apparatus 1 performs a recognition process asfollows so as to accurately and efficiently detect the marker 51.

(Recognition Process Using a Plurality of Different Precisions)

FIG. 5 is a diagram showing an object area used in the image recognitionprocess of the present embodiment. In the present embodiment, after acaptured image to be subjected to a recognition process is obtained, thegame apparatus 1 first performs a process (area detection process) ofdetecting an area (referred to as the “object area”) 54 of the capturedimage that includes the marker 51 (see the upper section of the tableshown in FIG. 5). The area detection process is performed on an entirearea 53 of the captured image and is performed with a lower precisionthan that of the marker detection process to be described later. Herein,the term “lower precision” means relatively increasing the intervalbetween pixels whose pixel values are to be referenced in the process.For example, in the present embodiment, the game apparatus 1 performsthe area detection process by referencing pixel values at a 3-pixelpitch across the captured image (i.e., by referencing pixel values ofpixels while skipping two pixels therebetween). Thus, since the processof detecting the object area 54 is performed with a lower precision, thedetection process is performed relatively quickly.

After the object area 54 is detected, the game apparatus 1 performs aprocess (marker detection process) of detecting the marker 51 from theimage of the object area 54. The marker detection process is performedonly on the object area 54 (rather than the entire area 53 of thecaptured image) and is performed with a higher precision than that ofthe area detection process described above (see the lower section ofFIG. 5). That is, in the marker detection process, pixels are referencedin a finer manner than the area detection process, i.e., the intervalbetween pixels whose pixel values are to be referenced is smaller thanthat of the area detection process. Specifically, in the presentembodiment, the game apparatus 1 performs the area detection process byreferencing the pixel values at a 1-pixel pitch across the capturedimage (i.e., by referencing the pixel value of every pixel). Thus, sincethe marker detection process is performed with a higher precision, theposition and direction of the marker 51 obtained as detection resultsare calculated with a high precision. Since the marker detection processis performed only on the object area 54, the game apparatus 1 canperform the process more quickly than when the process is performed onthe entire area 53 of the captured image.

As described above, in the present embodiment, the game apparatus 1first performs the process of detecting the object area 54, and thenperforms the process of detecting the marker 51 on the object area 54.Therefore, the process of detecting the marker 51 can be performed onlyon the required area (i.e., the object area), and the process ofdetecting the marker 51 can be performed in a short period of time.Since the process of detecting the object area 54 is performed with alower precision, the detection process can also be performed in a shortperiod of time, and it is therefore possible to perform the entirerecognition process in a short period of time. Moreover, since theprocess of detecting the marker 51 is performed with a higher precision,the position and direction of the marker 51 obtained as recognitionresults can be calculated with a high precision. That is, according tothe present embodiment, it is possible to quickly and precisely performthe recognition process of detecting the marker 51 from the capturedimage.

Note that different precisions are used within the marker detectionprocess in the present embodiment. That is, the marker detection processincludes a plurality of processes which are performed with differentprecisions. The details of these processes will be described later.

(Recognition Process of Detecting Marker Based on Center Position ofInner Graphic Pattern)

The recognition process of the present embodiment uses the marker 51with a graphic pattern thereon including a plurality of inner graphicpatterns, and detects the marker 51 by calculating the center positionsof the inner graphic patterns. FIG. 6 is a diagram showing the markerused in the present embodiment. As shown in FIG. 6, the marker 51includes a first graphic pattern 55 and a second graphic pattern 57. Thefirst graphic pattern 55 includes a plurality of (eight in FIG. 6) whitecircular graphic patterns 56 drawn therein. In the present embodiment,the white circular graphic patterns 56 are the inner graphic patterns.In the present embodiment, the eight inner graphic patterns 56 arelocated at the four vertices of a rectangular shape (square) and themiddle points of the four sides thereof. Note that the first graphicpattern may be any pattern with a plurality of graphic patterns (innergraphic patterns) drawn therein, and there is no limitation on theposition, shape, size and color of the inner graphic patterns. While thesecond graphic pattern 57 is an arrow-shaped graphic pattern in FIG. 6,the second graphic pattern 57 may be any graphic pattern.

Note that since the marker 51 shown in FIG. 6 is used in the presentembodiment, the game apparatus 1 detects the area of the first graphicpattern 55 of the marker 51 as the object area in the above-describedarea detection process. Then, in the above-described marker detectionprocess, the position and direction of the marker 51 are detected bydetecting the first graphic pattern 55 with a high precision.

Now, a method for detecting the outline of the first graphic pattern 55of the marker 51 in order to detect the position of the marker 51 willbe discussed. FIG. 7 is a diagram showing an image of the first graphicpattern 55 with the left and right portions of its outline blurred. Animage captured by a camera may sometimes be blurred and have an unclearoutline due to a so-called “camera shake”, etc. For example, if thecamera (the game apparatus 1) moves left and right when capturing animage, the outline will be blurred in the left-right direction to beunclear as shown in FIG. 7. Where the captured image is blurred and hasan unclear outline as described above, it is difficult to detect theoutline accurately by the outline detecting method described above, andit is therefore impossible to accurately calculate the position of thegraphic pattern. An outline can be detected based on the difference inpixel values (luminance values, color values, etc.) between pixels, forexample, but if the captured image is blurred, the detected position ofthe outline varies substantially depending on the content of the image,the threshold value, etc. For example, with the example of the imageshown in FIG. 7, a dotted line 58 shown in FIG. 7 may be detected as anoutline, or a dotted line 59 may be detected as an outline. In suchcases, the length of the first graphic pattern 55 in the lateraldirection is not calculated accurately, and the position of the firstgraphic pattern 55 cannot be calculated accurately. Thus, with a methodfor detecting an outline and calculating the position of the marker fromthe outline, it may not be possible to calculate the position of themarker accurately when the captured image is blurred.

Therefore, in the recognition process of the present embodiment, thegame apparatus 1 detects the marker 51 by detecting the inner graphicpatterns 56 of the first graphic pattern 55, and calculating the centerpositions of the inner graphic patterns 56. FIG. 8 shows the centerposition of the inner graphic pattern in a portion of the first graphicpattern 55. In the above-described marker detection process, the gameapparatus 1 detects the area (outline) of the inner graphic pattern 56,and calculates the center point P of the detected area (see FIG. 8).Now, referring to FIG. 8, because the captured image is blurred, eitherthe white dotted line or the black dotted line shown in FIG. 8 may bedetected as the outline, but the position of the center point P will bethe same in either case.

FIG. 9 is a diagram showing the center points of the inner graphicpatterns 56 in the first graphic pattern 55. In the present embodiment,the center points P1 to P8 of the inner graphic patterns 56 arecalculated. Herein, the length of the first graphic pattern 55 in theleft-right direction is represented by the length from the point P1 tothe point P3, for example. This length will be the same whether thewhite dotted line or the black dotted line shown in FIG. 8 is detectedas the outline. That is, this length can be calculated accurately evenif the captured image is blurred.

In the present embodiment, the game apparatus 1 identifies the positionsof four points on the marker 51 from the center points P1 to P8 of theinner graphic patterns 56, as will be described later in detail. Thepositions of the four points are the positions of the four vertices ofthe rectangular shape (square) represented by the eight inner graphicpatterns 56. The position and direction of the marker 51 are identifiedby the positions of the four points. Note however that the positions ofthe four points are the vertices of the square, and therefore take thesame values whether the marker 51 is not rotated or rotated by 90°, 180°or 270°. Therefore, with the positions of the four points, it is notpossible to identify the direction of the marker 51 with respect to thefour directions (up, down, left and right), and thus it is not possibleto uniquely identify the direction of the marker 51. Therefore, the gameapparatus 1 uniquely identifies the direction of the marker 51 bydetecting the second graphic pattern 57. Note that the method foridentifying the direction of the marker 51 using the second graphicpattern 57 will be described later.

As described above, in the present embodiment, the game apparatus 1calculates the center points P1 to P8 of the inner graphic patterns 56in the marker detection process. Then, the position of the marker 51 iscalculated by identifying the positions of the four points of the marker51 from the center points P1 to P8. Now, since the center points P1 toP8 are calculated accurately even where the captured image is blurred,the game apparatus 1 can accurately calculate the position of the marker51 by using the center points P1 to P8 even if the captured image isblurred.

[Details of Game Process]

Next, referring to FIGS. 10 to 15, the details of the game processexecuted by the game program will be described. First, various data usedin the game process will be described. FIG. 10 is a diagram showingvarious data to be used in processes performed by the game program. InFIG. 10, a game program 61, captured image data 62, and game processdata 63 are stored in the main memory 32 of the game apparatus 1.

The game program 61 is a program for instructing the CPU 31 of the gameapparatus 1 to execute a game process (FIG. 11) to be described later. Apart or whole of the game program 61 is read out from the memory card 29at an appropriate timing so as to be stored in the main memory 32. Thegame program 61 includes a program for executing the above-describedimage recognition process, in addition to a program for allowing thegame to progress.

The captured image data 62 is data representing an image (capturedimage) captured by a camera (the internal camera 23 or the externalcamera 25). As the captured image data 62, only the latest image dataobtained from the camera may be stored, or a predetermined number oflatest pieces of image data may be stored.

The game process data 63 represents various data to be used in the gameprocess. The game process data 63 includes object area data 64, outlinedata 65, inner area data 66, center position data 67, vertex data 68,and recognition result data 69. Note that in addition to those describedabove, the game process data 63 stores various data necessary for thegame, such as data of various objects (e.g., the cannon 52) appearing inthe game, and sound data such as BGM.

The object area data 64 is data representing the object area describedabove, i.e., the area of the first graphic pattern 55 in the capturedimage. The object area data 64 is obtained by the above-described areadetection process. Note that in the present embodiment, the detectionprocess is performed with a low precision in the area detection process,and therefore an area that is actually not the first graphic pattern 55may be detected as an area of the first graphic pattern 55. Thus, aplurality of areas may be detected as object areas in the area detectionprocess. In such a case, the object area data 64 for each area detectedis stored in the main memory 32 (i.e., a plurality of pieces of objectarea data are stored). Note that the object area data 64 may be any dataas long as it represents an area, and it may represent only the outlineof the area or may represent every pixel of the area.

The outline data 65 is data representing the outline of the firstgraphic pattern 55 in the captured image. The outline data 65 is datarepresenting an area of the first graphic pattern 55, as does the objectarea data 64, but the outline data 65 differs from the object area data64 in that it is detected with a higher precision.

The inner area data 66 is data representing an area of the inner graphicpattern 56 in the first graphic pattern 55. As described above, thefirst graphic pattern 55 includes a plurality of (eight) inner graphicpatterns 56. Therefore, if the eight inner graphic patterns 56 aredetected correctly in the marker detection process, eight pieces of theinner area data 66 are stored in the main memory 32. Note however thatthere may be cases in practice where the eight inner graphic patterns 56are not all detected. In such a case, a number of pieces of the innerarea data 66 equal to the number of inner graphic patterns 56 detectedare stored in the main memory 32. Note that the inner area data 66 maybe any data as long as it represents an area, and it may represent onlythe outline of the area or may represent every pixel of the area.

The center position data 67 is data representing the center positions ofthe inner graphic patterns 56. The center position data 67 represents anumber of center positions equal to the number of inner graphic patterns56 detected in the marker detection process.

The vertex data 68 is data representing the positions of the fourvertices of a rectangular shape (square) represented by the eight innergraphic patterns 56. The four vertices are calculated based on thecenter positions of the eight inner graphic patterns 56 as will bedescribed later in detail.

The recognition result data 69 is data representing the position anddirection of the marker 51, which is output as the result of therecognition process. The position and direction of the marker 51 arecalculated based on the four vertices represented by the vertex data 68,and the direction of the second graphic pattern 57 with respect to thefirst graphic pattern 55 in the captured image, as will be describedlater in detail.

Next, the details of the game process executed by the game apparatus 1will be described with reference to FIGS. 11 to 16. FIG. 11 is a mainflow chart showing the flow of a game process executed by the gameapparatus 1. When the power button 14F is pressed down and the power ofthe game apparatus 1 is turned ON, the CPU 31 of the game apparatus 1displays a menu screen (e.g., an image including images (icons)representing various applications) used for instructing to start variousapplications. On the menu screen, the user instructs to start the gameprogram 61, e.g., instructs to select the icon of the game program 61.When the start of the game program 61 is instructed on the menu screen,the CPU 31 initializes the main memory 32, etc., and starts executingthe game program 61 for performing the process shown in FIG. 11. In thepresent embodiment, as the game program 61 is executed, the CPU 31functions as various sections recited in the claims. That is, the gameprogram 61 instructs the CPU 31 to function as various sections recitedin the claims.

First, in step S1, the CPU 31 determines whether a captured image hasbeen obtained. That is, it determines whether data of a captured imagehas been sent from a camera (the external camera 25 in the presentembodiment). If data of a captured image has been sent from the camera,the CPU 31 stores the data as the captured image data 62 in the mainmemory 32. Note that it is assumed in the present embodiment that whilethe process loop through steps S1 to S7 is executed once every frameperiod ( 1/60 sec), the camera sends a captured image with a lowerfrequency (once every about a few frame periods). Note however that acaptured image may be obtained once every frame period (or every periodthat is shorter than a frame period) in other embodiments. If thedetermination result of step S1 is affirmative, the process of step S2is performed. On the other hand, if the determination result of step S1is negative, the process of steps S2 to S4 is skipped, and the processof step S5 to be described below is performed.

In step S2, the CPU 31 executes a recognition process on the capturedimage obtained in step S1. In the recognition process, it is determinedwhether the image of the marker 51 is included in the captured image,and if the image of the marker 51 is included therein, the direction andposition of the marker 51 in the captured image are calculated. Now,referring to FIG. 12, the details of the recognition process will bedescribed.

FIG. 12 is a flow chart showing the flow of the recognition process(step S2) shown in FIG. 11. In the recognition process, first, in stepS11, the CPU 31 executes the above-described area detection process,i.e., detects an area (object area) including the first graphic pattern55. As described above, the area detection process of step S11 isperformed with a low precision. Specifically, the CPU 31 detects theobject area from the captured image based on pixel values obtained at a3-pixel pitch across the captured image. More specifically, the CPU 31reads out the captured image data 62 from the main memory 32, andextracts pixel values at a 3-pixel pitch from the pixel values of thepixels represented by the image data. Then, the object area is detectedbased on the extracted pixel values. FIG. 13 is a diagram showing pixelsin the captured image whose pixel values are referenced in the processof step S11. In FIG. 13, hatched pixels are pixels whose pixel valuesare to be referenced.

In the present embodiment, the object area is detected based on thedifference in pixel values (the luminance values in the presentembodiment, but color values may also be used) between adjacent pixels.That is, in the present embodiment, since a white frame is formed aroundthe black first graphic pattern 55 (see FIG. 6), it is possible todetect pixels along the outline portion of the first graphic pattern 55by detecting pixels whose luminance values are lower than adjacentpixels by a predetermined value or more. Note that the term “adjacentpixels” means adjacent ones of pixels whose pixel values are referenced.In the example of FIG. 13, “adjacent pixels” are one hatched pixel andanother pixel (also hatched) that is at a 3-pixel pitch from the firstpixel. After detecting pixels along the outline portion, the CPU 31combines adjacent ones of the detected pixels into a single collection,thus identifying a single object area. Data representing pixels formingthe identified object area is stored in the main memory 32 as a piece ofobject area data 64 for each object area. Note that where an identifiedobject area is too large or too small, the CPU 31 may exclude the objectarea so that the object area data 64 representing the object area is notstored. The process of step S12 is performed following step S11.

Note that the method for detecting the object area in step S11 may beany method. For example, in other embodiments, the CPU 31 may detect theoutline of the first graphic pattern 55 by binarizing the pixel valueswith a predetermined threshold value, and determine, as an object area,the area within the detected outline. In other embodiments, the CPU 31may detect the object area by a pattern matching method. That is, apattern image of the first graphic pattern 55 may be pre-stored in themain memory 32, and an area that is identical (similar) to the patternimage is detected from within the area of the captured image so as todetect the object area. In such a case, the pattern matching process maybe similar to that of a conventional augmented reality technique.

While the CPU 31 detects an area of the first graphic pattern 55 as theobject area in step S11 in the present embodiment, an area of the marker51 including the first graphic pattern 55 and the second graphic pattern57 may be detected as the object area in other embodiments.

In the present embodiment, the object area is detected with a lowprecision as described above, and the object area detection process isbased on the luminance difference between the first graphic pattern 55of the marker 51 and the surrounding white frame. Therefore, if thewhite frame of the marker 51 is too narrow, the white frame may notappear in the captured image with a low precision, thereby failing toaccurately detect the object area. Therefore, it is preferred that thewhite frame of the marker 51 has a sufficient thickness so that thewhite frame appears on the captured image even with a low precision.Note that the thickness of the white frame can be determinedappropriately according to the expected range of distance from thecamera to the marker 51 during the use of the game apparatus 1, theresolution of the captured image, the sizes of the marker 51, etc.

In step S12, the CPU 31 determines whether the object area has beendetected in step S11. If the object area is not detected in step S11,the process of step S13 is performed. On the other hand, if one or moreobject areas are detected in step S11, the process of step S14 isperformed.

In step S13, the CPU 31 determines that the recognition of the marker 51has failed. That is, the CPU 31 stores data indicating the recognitionfailure in the main memory 32 as the recognition result data 69. Afterstep S13, the CPU 31 exits the recognition process.

In step S14, the CPU 31 selects one of object areas detected in stepS11. Note that where a plurality of object areas are detected in stepS11, the process step through steps S14 to S16 is repeated. In such acase, in step S14, an object area that has not been selected in theprocess step is selected. The process of step S15 is performed followingstep S14.

In step S15, the CPU 31 performs the marker detection process on theobject area selected in step S14. The marker detection process is aprocess of detecting the marker 51 from the object area, and calculatingthe position and direction of the marker 51. Referring now to FIG. 14,the details of the marker detection process will be described.

FIG. 14 is a flow chart showing the flow of the marker detection process(step S15) shown in FIG. 12. In the marker detection process, first, instep S21, the CPU 31 extracts the outline of the first graphic pattern55 with an intermediate precision. Herein, the term “intermediateprecision” means a precision that is higher than that used in theprocess of step S11, but is lower than that used in the process of stepsS23 and S25 to be described later. That is, in the present embodiment,the marker detection process is performed with two levels of precision,and the process of step S21 is performed with the lower precision. Thedetails of the process of step S21 will now be described.

While the process of step S21 is performed within the object areadetected in step S11, the process of step S11 and that of step S21 areperformed with different precisions. Therefore, in step S21, the CPU 31first determines an area to be the subject of the process of step S21,which corresponds to the object area detected in step S11. That is, anarea to be the subject of the process of step S21 is determined frompixels represented by the object area data 64 obtained in step S11. Morespecifically, the CPU 31 associates each pixel (each hatched pixel shownin FIG. 13) to be referenced in the low-precision process (step S11)with eight pixels surrounding the pixel (pixels delimited by aone-dot-chain line in FIG. 13). Then, an area including pixels withinthe object area detected in step S11 and sets of surrounding eightpixels associated therewith is determined as an area to be the subjectof the process of step S21. FIG. 15 is a diagram showing an example ofan area to be the subject of the process of step S21. FIG. 15 assumes acase where pixels Q1 to Q3 are detected in step S11, and an area on theright of the pixels Q1 to Q3 is detected as the object area. In such acase, the area A1 on the right of the one-dot-chain line shown in FIG.15 is the area to be the subject of the process of step S21.

After an area to be the subject of the process is determined, the CPU 31extracts the outline of the first graphic pattern 55 with anintermediate precision. In the present embodiment, the outlineextracting process is performed based on pixel values obtained at a2-pixel pitch across the captured image. FIG. 16 is a diagram showingpixels in the captured image whose pixel values are referenced in theprocess of step S21. For example, in a case where the area A1 shown inFIG. 15 is the subject of the process of step S21, pixel values ofhatched pixels shown in FIG. 16 are referenced in the outline extractingprocess.

Note that the outline extracting process of step S21 may be performed bya method using the luminance value difference, as is the method fordetecting the object area in step S11, or may be performed by anothermethod. Data of the pixels forming the outline extracted in step S21 isstored in the main memory 32 as the outline data 65. The process of stepS22 is performed following step S21.

In step S22, the CPU 31 determines whether the outline extracted in stepS21 represents the first graphic pattern 55. In the present embodiment,the CPU 31 makes the determination of step S22 based on whether theoutline represents a rectangular shape. That is, the CPU 31 reads outthe outline data 65 from the main memory 32, and approximates the pixelsforming the outline by straight lines, thus calculating a polygon formedby the pixels. Then, it is determined whether the outline represents thefirst graphic pattern 55 based on whether the calculated polygon isrectangular. If the determination result of step S22 is affirmative, theprocess of step S23 is performed. On the other hand, if thedetermination result of step S22 is negative, the process of step S30 tobe described later is performed.

Note that while the determination whether the extracted outlinerepresents the first graphic pattern 55 is made based on whether theoutline is rectangular in step S22, the determination may be made byanother method. For example, in other embodiments, where the size of themarker 51 is known, the determination may be performed based on, forexample, whether the size (e.g., the area or the circumferential length)of the extracted outline is greater than or equal to a predeterminedvalue.

In step S23, the CPU 31 detects an area of the inner graphic pattern 56with a high precision. Herein, the term “high precision” means aprecision that is higher than those used in the process of step S11 andthe process of step S21. In the present embodiment, the CPU 31 detectsthe area of the inner graphic pattern 56 by referencing pixel values ata 1-pixel pitch, i.e., by referencing pixel values of all pixels withinan area to be the subject of the process of step S23.

Although the process of step S23 is performed across the arearepresented by the outline extracted in step S21, the precision isdifferent between the process of step S21 and the process of step S23.Therefore, the area to be the subject of the process needs to bedetermined in step S23, as in step S21. The area to be the subject ofthe process may be determined by a method similar to that of step S21.That is, the CPU 31 associates each reference pixel that can bereferenced in the intermediate-precision process (step S21) with threeadjacent pixels (e.g., one on the right, one below, and one on lowerright of the reference pixel). Then, the reference pixel included in theoutline extracted in step S21 and the three pixels associated with thereference pixel are determined to be the subject of the process of stepS23.

After the area to be the subject of the process is determined, the CPU31 detects the area of the inner graphic pattern 56 with a highprecision. Herein, the area of the inner graphic pattern 56 may bedetected by a method which is based on the luminance value difference,as is the method for detecting the object area in step S11, or may bedetected by another method. Note that where a method based on theluminance value difference is used, the CPU 31 preferably detects apixel whose luminance value is higher than that of an adjacent pixel bya predetermined value or more since the inner graphic pattern 56 to bedetected in step S23 is white with the surrounding area being black. Inthe present embodiment, the CPU 31 detects (a plurality of) pixelsrepresenting the outline of the inner graphic pattern 56 as informationrepresenting the area of the inner graphic pattern 56. Data of pixelsrepresenting the area of the inner graphic pattern 56 detected in stepS23 is stored in the main memory 32 as the inner area data 66. Where aplurality of inner graphic patterns 56 are detected in step S23, onepiece of inner area data 66 is stored in the main memory 32 for eachinner graphic pattern. The process of step S24 is performed followingstep S23 described above.

Note that in other embodiments, the CPU 31 may set a condition regardingthe shape and/or size of the detected area so as to exclude areas thatdo not satisfy the condition in step S23. That is, the CPU 31 may notstore the inner area data 66 for those of areas detected based on theluminance difference whose shape is not circular or whose size is notwithin a predetermined range.

In step S24, the CPU 31 determines whether a predetermined number ofareas have been detected in step S23. Herein, the predetermined numberis the number of inner graphic patterns 56 drawn within the firstgraphic pattern 55, i.e., eight. Specifically, the CPU 31 determineswhether the number of pieces of the inner area data 66 stored in themain memory 32 is eight. If the determination result of step S24 isaffirmative, the process of step S25 is performed. On the other hand, ifthe determination result of step S24 is negative, the process of stepS30 to be described later is performed.

In step S25, the CPU 31 calculates the center position of the innergraphic pattern 56 with a high precision. The center position of theinner graphic pattern 56 is calculated based on the area detected instep S23, i.e., based on the pixels representing the outline of theinner graphic pattern 56. The specific method for calculating the centerposition may be any method, and the center position can be obtained bycalculating the average value among the position coordinates of thepixels representing the outline of the inner graphic pattern 56, forexample. In the present embodiment, the CPU 31 reads out the inner areadata 66 from the main memory 32, and calculates the average value amongthe pixels represented by the inner area data 66. Note that eight piecesof the inner area data 66 are stored, and the process of calculating theaverage value is performed for each piece of data. The data representingthe eight average values calculated is stored in the main memory 32 asthe center position data 67. The process of step S26 is performedfollowing step S25.

In step S26, the CPU 31 calculates the positions of the four verticesrepresenting the position of the marker 51 based on the center positionscalculated in step S25. Herein, the four vertices are the vertices of arectangular shape of which the vertices and the middle points of thesides are represented by the eight inner graphic patterns 56. Referringnow to FIGS. 17 and 18, a method for calculating the four vertices willbe described.

FIG. 17 is a diagram showing an example of a captured image representinga first graphic pattern. In FIG. 17, the points P1 to P8 are pointsrepresenting the positions calculated in step S26 as the centerpositions of the eight inner graphic patterns 56. In step S26, the CPU31 first calculates four straight lines L1 to L4 passing through threeof the eight center positions. Specifically, the CPU 31 calculates thestraight line L1 passing through the points P1 to P3, the straight lineL2 passing through the points P1, P4 and P6, the straight line L3passing through the points P6 to P8, and the straight line L4 passingthrough the points P3, P5 and P8. The straight lines L1 to L4 can eachbe calculated as an approximate straight line passing through threepoints by using the least squares method, etc.

FIG. 18 is a diagram showing four vertices P11 to P14 calculated fromthe straight lines L1 to L4 shown in FIG. 17. After the straight linesL1 to L4 are calculated, the CPU 31 calculates the vertices P11 to P14of the rectangular shape formed by the straight lines L1 to L4. That is,the CPU 31 calculates the intersection P11 between the straight line L1and the straight line L2, the intersection P12 between the straight lineL1 and the straight line L4, the intersection P13 between the straightline L2 and the straight line L3, and the intersection P14 between thestraight line L3 and the straight line L4. Thus, the four vertices P11to P14 can be calculated. Data representing the calculated positions ofthe four vertices is stored in the main memory 32 as the vertex data 68.The process of step S27 is performed following step S26.

Calculating the four vertices in step S26 means that positions of fourpoints in the marker 51 have been calculated. Now, the position anddirection of the marker 51 can be calculated based on the positions ofthe four points in the marker 51. Note however that the positions of thefour points are the vertices of the square, and therefore take the samevalues whether the marker 51 is not rotated or rotated by 90°, 180° or270°. Therefore, with the positions of the four points, it is notpossible to identify the direction of the marker 51 with respect to thefour directions (up, down, left and right), and thus it is not possibleto uniquely identify the direction of the marker 51. Therefore, the CPU31 uniquely identifies the direction of the marker 51 by identifying thedirection of the marker 51 with respect to the four directions (up,down, left and right) through the process of steps S27 to S29 to bedescribed below. Note that the CPU 31 may omit the process of steps S27to S29 to be described below and use the positions of the four points asthe recognition results if the direction from which an image of themarker 51 is to be captured is predetermined (known) among the fourdirections (up, down, left and right), or if the direction of the marker51 with respect to the four directions (up, down, left and right) isirrelevant (i.e., if it is considered the same whether the marker 51 isnot rotated or rotated by 90°, 180° or 270°).

In step S27, the CPU 31 detects the second graphic pattern 57 from anarea around the first graphic pattern 55. In the present embodiment, theCPU 31 detects the second graphic pattern 57 from one of the fourpredetermined areas located next to a rectangular shape formed by thefour vertices calculated in step S26 in the four directions (up, down,left and right). Since the second graphic pattern 57 is located next tothe first graphic pattern 55 (see FIG. 6), and the position of the firstgraphic pattern 55 is identified by the four vertices, the secondgraphic pattern 57 can be detected by performing a detection process forthe four predetermined areas.

FIG. 19 is a diagram showing an example of a predetermined area set instep S27. In FIG. 19, an area α delimited by the straight lines L5 to L8is the predetermined area. The predetermined area α is set along theside P11-P12 among the four sides of the rectangular shape formed by thefour vertices P11 to 914. The straight lines L5 to L8 are the lowerside, the upper side, the right side and the left side, respectively, ofthe predetermined area α where the upper side is the side of therectangular shape formed by the four vertices P11 to P14 on which thepredetermined area α is located. The straight lines L5 to L8, which arethe four sides of the predetermined area α, can be set as follows, forexample. The straight line L5, which is the lower side, is set parallelto the straight line L1 and apart from the straight line L1 by apredetermined first distance in a direction perpendicular to thestraight line L1. The straight line L6, which is the upper side, is setparallel to the straight line L1 and apart from the straight line L1 bya predetermined second distance (greater than the first distance) in adirection perpendicular to the straight line L1. The straight line L7,which is the right side, is set parallel to the straight line L4, whichis the right side of the rectangular shape formed by the vertices P11 toP14, and apart from the straight line L4 by a predetermined thirddistance in a direction of the straight line L1. The straight line L8,which is the left side, is set parallel to the straight line L2, whichis the left side of the rectangular shape formed by the vertices P11 toP14, and apart from the straight line L2 by a predetermined fourthdistance in the direction of the straight line L1. The predeterminedarea α can be calculated as described above. The other threepredetermined areas can be calculated as is the predetermined area α.

While the method for detecting the second graphic pattern 57 from one ofthe four predetermined areas may be any method, the second graphicpattern 57 is detected by a pattern matching method in the presentembodiment. That is, a pattern image of the second graphic pattern 57 ispre-stored in the main memory 32, and the CPU 31 detects an area that isidentical (similar) to the pattern image from the predetermined areas,thus detecting the second graphic pattern 57. The process of detectingthe second graphic pattern 57 may be performed with any precision. Notethat since this process can be performed with the limited areas, i.e.,the predetermined areas, it does not need to be performed with a highprecision.

As a specific process of step S27, the CPU 31 first reads out the vertexdata 68 from the main memory 32, and identifies the four predeterminedareas based on the four vertices. Next, the pattern image of the secondgraphic pattern 57 is read out from the main memory 32, and a portionthat is identical to the pattern image is detected from the fourpredetermined areas identified. The process of step S28 is performedfollowing step S27.

In step S28, the CPU 31 determines whether the second graphic pattern 57has been detected in step S27. That is, it is determined whether thereis an area, among the four predetermined areas, where the second graphicpattern 57 has been detected. If the determination result of step S28 isaffirmative, the process of step S29 is performed. On the other hand, ifthe determination result of step S28 is negative, the process of stepS30 to be described later is performed.

In step S29, the CPU 31 identifies the direction of the marker 51 (withrespect to the four directions (up, down, left and right)). Now, oncethe direction of the predetermined area where the second graphic pattern57 has been detected as viewed from the rectangular shape formed by fourvertices (the direction of the predetermined area as viewed from therectangular shape) is known, it means that the direction in which thesecond graphic pattern 57 is located with respect to the first graphicpattern 55 in the captured image is known, and it is therefore possibleto identify the direction of the marker 51. That is, in step S29, thedirection of the marker 51 is identified based on the direction in whichthe second graphic pattern 57 is located with respect to the firstgraphic pattern 55 in the captured image.

Note that in the present embodiment, the position and direction of themarker 51 are represented by two vectors V1 and V2 connecting togetherthe four vertices. Specifically, the vector V1 is a vector whose startpoint is the upper left vertex and whose end point is the upper rightvertex where it is assumed that the upper direction is the direction inwhich the second graphic pattern 57 is located with respect to the firstgraphic pattern 55. The vector V2 is a vector whose start point is thelower left vertex and whose end point is the lower right vertex, underthe above settings. Where the vectors V1 and V2 are assumed as describedabove, if the predetermined area where the second graphic pattern 57 hasbeen detected is located along the straight line L1 shown in FIG. 19,for example, the straight line L1 shown in FIG. 19 is the upper side ofthe rectangular shape formed by four vertices. Therefore, in such acase, the vector V1 is calculated as a vector whose start point is thepoint P11 and whose end point is the point P12, and the vector V2 iscalculated as a vector whose start point is the point P13 and whose endpoint is the point P14. The position and direction of the marker 51 canbe represented by these vectors V1 and V2.

In step S29, the CPU 31 calculates the vectors V1 and V2 by using thefour vertices so that they extend in a direction according the directionof the predetermined area where the second graphic pattern 57 has beendetected with respect to the rectangular shape formed by four vertices.Data representing the calculated vectors V1 and V2 is stored in the mainmemory 32. After step S29 described above, the CPU 31 exits the markerdetection process.

On the other hand, in step S30, the CPU 31 determines that the firstgraphic pattern 55 is absent in the object area which is the subject ofthe current marker detection process. In such a case, the various data65 to 68 calculated in steps S21 to S28 of the current marker detectionprocess are deleted from the main memory 32. After step S30, the CPU 31exits the marker detection process.

In the marker detection process, the CPU 31 detects a plurality of innergraphic patterns from the image of the object area (step S23) andcalculates the center positions of the inner graphic patterns, therebydetecting the marker 51 (steps S25 and S26). Thus, because the centerpositions of the inner graphic patterns are used, the position of themarker 51 can be calculated accurately even if the captured image isblurred.

By the series of processes of steps S21 to S23, the outline of the firstgraphic pattern 55 is extracted from within the object area (step S21),and it is determined whether the extracted outline represents the firstgraphic pattern 55 (step S22). Then, if the extracted outline representsthe first graphic pattern 55, a plurality of inner graphic patterns 56are detected from within the area corresponding to the outline (stepS23). That is, the inner graphic patterns 56 are detected from withinthe area corresponding to the outline that is determined to berepresenting the first graphic pattern 55. Thus, object areas that donot include the first graphic pattern 55 can be excluded before theprocess of detecting the inner graphic patterns 56, and it is thereforepossible to prevent the process of detecting the inner graphic patterns56 from being performed wastefully, and to efficiently perform therecognition process. Moreover, in the present embodiment, the outlineextracting process can be performed more quickly than the process ofdetecting the inner graphic pattern 56, and the game apparatus 1 cantherefore perform the recognition process in a shorter period of time.

By the series of processes of steps S23 to S25, areas of a plurality ofinner graphic patterns 56 are detected in the object area (step S23),and it is determined whether the number of areas detected is equal to apredetermined number (step S24). If the number of areas detected isequal to the predetermined number, the center positions of the areas arecalculated (step S25). That is, the center positions of the innergraphic patterns 56 are calculated for the object area where apredetermined number of areas have been detected. Thus, object areasthat do not include the first graphic pattern 55 can be excluded beforethe process of calculating the center positions of the inner graphicpatterns 56, and it is therefore possible to prevent the process ofcalculating the center positions from being performed wastefully, and toefficiently perform the recognition process.

In the marker detection process, the precision of the process ofdetecting the inner graphic patterns 56 (step S23) is equal to theprecision of the process of calculating the center positions of theinner graphic patterns 56 (step S25). In other embodiments, the CPU 31may perform the process of detecting the inner graphic patterns 56 witha lower precision than that of the process of calculating the centerpositions of the inner graphic patterns 56. That is, the pixel pitch atwhich pixel values are obtained in the process of detecting the innergraphic patterns 56 may be set to be larger than that at which pixelvalues are obtained in the center position calculating process. Then,the process of detecting the inner graphic patterns 56 can be performedmore quickly, and the recognition process can be performed more quickly.Note that the process of detecting the inner graphic patterns 56 may beperformed with the same precision as that of the process of extractingthe outline of the first graphic pattern 55 (step S21), or may beperformed with a higher precision than that of the extracting process.

Referring back to FIG. 12, the process of step S16 is performedfollowing the marker detection process of step S15. In step S16, the CPU31 determines whether the marker detection process has been performedfor all object areas detected in step S11. If the determination resultof step S16 is affirmative, the process of step S17 is performed. On theother hand, if the determination result of step S16 is negative, theprocess of step S14 is executed again. Thereafter, the process of stepsS14 to S16 is repeated until it is determined in step S16 that themarker detection process has been performed for all object areas. Thus,the marker detection process is performed for each object area detectedin step S11.

In step S17, the CPU 31 determines whether the marker 51 has beendetected in the marker detection process of step S15. Specifically, theCPU 31 determines whether the marker 51 has been detected based onwhether the recognition result data 69 is stored in the main memory 32.If the determination result of step S17 is affirmative, the process ofstep S18 is performed. On the other hand, if the determination result ofstep S17 is negative, the process of step S13 is executed again. Thatis, if the determination result of step S17 is negative, it isdetermined that the recognition of the marker 51 has failed, and theprocess exits the recognition process.

In step S18, the CPU 31 stores information representing the position anddirection of the marker 51 obtained in the marker detection process ofstep S15 as the recognition result. That is, data representing thevectors V1 and V2 calculated in step S29 of the marker detection processis stored in the main memory 32 as the recognition result data 69. Afterstep S18, the CPU 31 exits the recognition process.

Note that in the marker detection process, the vectors V1 and V2 may becalculated for each of a plurality of object areas detected in step S11from one captured image. That is, a plurality of sets of the vectors V1and V2 may be stored in the main memory 32 in a single recognitionprocess. In such a case, one of the sets represents the correct marker51, and the others are resulting from erroneous detection. Therefore, insuch a case, the CPU 31 may select the vectors V1 and V2 which representthe correct marker 51 based on the shape, the arrangement, etc., of thegraphic patterns (the first graphic pattern 55, the inner graphicpattern 56 and the second graphic pattern 57) obtained in the markerdetection process. For example, it is possible to select the correctvectors V1 and V2 based on the outline of the first graphic pattern 55extracted in step S21, the arrangement of the inner graphic patterns 56detected in step S23, the degree of similarity between the secondgraphic pattern 57 detected in step S27 and the pattern image, etc. TheCPU 31 stores data representing the selected vectors V1 and V2 in themain memory as the recognition result data 69. Thus, even if a pluralityof sets of the vectors V1 and V2 are obtained (by erroneous detection)in the recognition process, it is possible to obtain correct recognitionresults. Note that in other embodiments, if a plurality of sets of thevectors V1 and V2 are obtained in a single recognition process, the CPU31 may determine that the recognition has failed.

In the recognition process described above, an object area is detected(step S11) based on pixel values obtained at a first pitch (3-pixelpitch) across the captured image. Then, the first graphic pattern 55 isdetected from the image of the object area based on pixel valuesobtained at a second pitch (1-pixel or 2-pixel pitch) which is smallerthan the first pitch across the object area of the captured image. Thus,it is possible to perform the recognition process quickly by performingthe process of detecting the object area in a short period of time.Since the process of detecting the first graphic pattern 55 is performedwith a high precision, it is possible to detect the first graphicpattern 55 accurately with a high precision. That is, with therecognition process described above, the process of detecting the firstgraphic pattern 55 from the captured image can be performed quickly andprecisely.

Note that it is assumed in the present embodiment that the recognitionprocess of step S2 is performed in a frame period, and the recognitionprocess is completed within the process loop through steps S1 to S7.Depending on the resolution of the captured image or the capacity of theCPU 31, etc., it may be difficult for a single recognition process to becompleted within a frame period. Therefore, in other embodiments, therecognition process does not have to be performed with the same cycle asa process such as the display process (step S5) which is performed witha cycle of a frame period. That is, only a portion of the recognitionprocess may be performed in a single iteration of step S2. In such acase, the amount of process to be performed in each iteration of step S2is adjusted so that the series of processes of steps S1 to S7 can becompleted within a frame period. In other words, the recognition processmay be performed separately from, and in parallel to, the process loopthrough steps S1 to S7 (excluding S2), and may be performed while theCPU 31 is idling.

Referring back to FIG. 11, the process of step S3 is performed followingthe recognition process of step S2. In step S3, the CPU 31 determineswhether the recognition process of step S2 has succeeded. Thedetermination of step S3 can be made based on whether the recognitionresult data 69 stored in the main memory 32 represents the position anddirection of the marker 51. If the determination result of step S3 isaffirmative, the process of step S4 is performed. On the other hand, ifthe determination result of step S3 is negative, the process of step S4is skipped and the process of step S5 to be described later isperformed.

In step S4, the CPU 31 calculates the position and orientation of avirtual camera based on the recognition results of step S2. The virtualcamera is a camera assumed in the virtual space for determining thepoint of view, the viewing direction and the viewing angle for producingan image of the virtual space. In step S4, the CPU 31 first calculatesthe positional relationship between the game apparatus 1 (camera) andthe marker 51 based on the position and direction of the marker 51 whichare the recognition results of step S2. The positional relationship isrepresented as the three-dimensional position and orientation of one ofthe game apparatus 1 and the marker 51 with respect to the other, forexample. Next, the CPU 31 calculates the position and orientation of thevirtual camera in the virtual space based on the positionalrelationship. The position and orientation of the virtual camera arecalculated so that the positional relationship between the virtualcamera in the virtual space and the position corresponding to the marker51 in the virtual space is equal to the positional relationship betweenthe game apparatus 1 and the marker 51 in the real space. Note that theprocess of step S4 may be a method similar to a process used in aconventional augmented reality technique as described in, for example,Non-Patent Document 1 identified above. If the recognition process ofstep S2 has failed, the CPU 31 does not calculate the position andorientation of the virtual camera, but uses the position and orientationof the virtual camera which were calculated when the recognition processsucceeded last. The process of step S5 is performed following step S4.

In step S5, the CPU 31 performs a game control process. The game controlprocess is a process for allowing the game to progress by, for example,moving an object around in the virtual space according to, for example,an input from a user (player). Specifically, the game control processincludes the process of controlling the action of the player characteraccording to the user input, and the process of controlling the actionof the object (e.g., the cannon 52 shown in FIG. 4) according to controlrules prescribed in the game program 61. The CPU 31 may use, as gameinputs, the recognition result data 69 as well as operations on variousinput devices (the touch panel 13 and the operation buttons 14A to 14L)provided on the game apparatus 1. For example, the position of thevirtual camera calculated from the recognition result data 69 may beused as the position of the player character. The process of step S6 isperformed following step S5.

In step S6, the CPU 31 produces a synthesized image by synthesizingtogether the captured image and the image (virtual image) of the virtualobject in the virtual space, and displays the synthesized image on adisplay apparatus (the upper LCD 22). That is, the CPU 31 first producesa virtual image based on the position and orientation of the virtualcamera set in step S4. Thus, there is produced an image of the virtualobject as viewed from the position of the virtual camera. Next, the CPU31 produces and displays a synthesized image by synthesizing theproduced image of the virtual object with the captured image. Note thatthe captured image used for producing the synthesized image may be thelatest captured image obtained or the latest captured image for whichthe recognition process has succeeded. Note that the synthesized imageproducing process may be a process similar to that of a conventionalaugmented reality technique. The process of step S7 is performedfollowing step S6.

In step S7, the CPU 31 determines whether the game should be ended. Thedetermination of step S7 is made based on, for example, whether the gamehas been cleared, the game has been over, the player has given aninstruction to quit the game, etc. If the determination result of stepS7 is negative, the process of step S1 is executed again. The processloop through steps S1 to S7 is repeated until it is determined in stepS7 that the game should be ended. On the other hand, if thedetermination result of step S7 is affirmative, the CPU 31 ends the gameprocess shown in FIG. 11. The game process is as described above.

With the game process described above, it is possible to perform therecognition process quickly and precisely by changing the processprecision between the object detection process (step S11) of detectingthe area of the first graphic pattern 55 from the captured image, andthe marker detection process (step S15) of detecting the marker 51including the first graphic pattern 55 from the detected area. Bycalculating the center positions of the inner graphic patterns 56 in thefirst graphic pattern 55 and calculating the positions (four vertices)on the marker 51 based on the center positions, it is possible toaccurately calculate the position of the marker 51 even if the capturedimage is blurred.

Note that by the series of processes of steps S3 to S6, if the marker 51is detected (Yes in step S3), the positional relationship between themarker 51 and the external camera 25 is calculated based on the processresult of the recognition process (the detection result for the marker51) (step S4). Then, the virtual image representing the virtual objectplaced in the virtual space is produced based on the positionalrelationship, and an image obtained by synthesizing the virtual imagewith the captured image is displayed on the display apparatus (step S5).Therefore, in the present embodiment, the recognition process describedabove can be used in a so-called “augmented reality technique”. Herein,if an augmented reality technique is used as described in the presentembodiment, it is preferred that the position and direction of themarker 51 are calculated accurately in the recognition process. In agame process such as one of the present embodiment, it is preferred thatthe recognition process is performed quickly as it is undesirable if aprocess delay deteriorates the response speed to game operations ordelays the game progress. Therefore, the recognition process of thepresent embodiment, which can be performed quickly and precisely, isparticularly effective in a game process using an augmented realitytechnique.

[Variations]

The above embodiment is an example of how the present invention can becarried out, and the present invention may be carried out as describedbelow, for example, in other embodiments.

(Variation Regarding Method for Calculating Four Vertices from CenterPositions)

In step S24, the CPU 31 determines whether the number of areas of theinner graphic patterns 56 detected in step S23 is a predeterminednumber. Herein, in the present embodiment, the four vertices areidentified by calculating four straight lines from the center positionsof the eight inner graphic patterns 56, and it is therefore possible tocalculate the four vertices even if the eight inner graphic patterns 56have not all been detected. For example, the four straight lines L1 toL4 can be calculated even if one of the center points P1 to P8 ismissing in FIG. 17. Therefore, in other embodiments, the CPU 31 maydetermine in step S24 whether the number of areas of the inner graphicpatterns 56 detected in step S23 is within a predetermined range.Herein, the predetermined range is a range including the number of innergraphic patterns (eight), and is, for example, a range from seven toeight.

Note that in the above embodiment, when calculating four straight linesfrom seven center positions, it is necessary to identify one straightline from two center positions. In this case, it is necessary to avoid asituation where an invalid straight line is calculated from an invalidset of two center positions (e.g., a case where a straight line passingthrough the center point P3 and the center point P4 is calculated).Therefore, the CPU 31 may calculate the four straight lines from sevencenter positions so as to satisfy the following conditions, for example.

Condition 1: To pass through two or three of the seven center positions.

Condition 2: To lie generally parallel to one of four straight linesthat represent the outline of the first graphic pattern 55.

Condition 3: To lie within a predetermined distance from one of fourstraight lines that represent the outline of the first graphic pattern55 (to not pass near the center of the first graphic pattern 55).

Note that in Conditions 2 and 3, the four straight lines that representthe outline of the first graphic pattern 55 may be those calculated instep S22. By calculating straight lines each satisfying Conditions 1 to3, it is possible to calculate four straight lines from seven centerpositions.

As described above, the image object may be the first graphic pattern 55including three or more inner graphic patterns arranged along a singlestraight line, and an approximate straight line may be calculated fromthe center positions of two or more inner graphic patterns so as tocalculate the position of the marker 51 (the positions of the fourvertices) based on the approximate straight line. Then, there isprovided redundancy in the number of inner graphic patterns, whereby itis possible to calculate the positions of the four vertices even if theinner graphic patterns cannot all be detected.

(Variation Regarding Marker)

While the above embodiment is directed to an example where the marker 51including the first graphic pattern 55 with the eight inner graphicpatterns 56 drawn therein is used, the marker that can be used in thepresent invention is not limited to this. For example, in otherembodiments, the number of the inner graphic patterns 56 is not limitedto eight. FIG. 20 is a diagram showing a marker of a variation of thepresent embodiment. A marker 73 shown in FIG. 20 includes a firstgraphic pattern 74 with twelve inner graphic patterns 56 therein, and asecond graphic pattern 57 similar to that of FIG. 6. The twelve innergraphic patterns 56 in the first graphic pattern 74 are arranged at thefour vertices and along the four sides of the rectangular shape(square). In other embodiments, the marker 73 shown in FIG. 20 may beused instead of the marker 51 shown in FIG. 6. In a case where themarker 73 is used, the CPU 31 may calculate the straight lines L1 to L4from three or four center positions in step S26 described above. Then,there is provided redundancy in the number of inner graphic patterns,whereby the CPU 31 can calculate the positions of the four vertices evenif the inner graphic patterns cannot all be detected.

While it is assumed in the above embodiment that the inner graphicpatterns 56 are all of the same color (white), the colors of the innergraphic patterns 56 may differ from one another depending on theposition within the first graphic pattern 55 in other embodiments. FIG.21 is a diagram showing a marker of another variation of the presentembodiment. A marker 75 shown in FIG. 21 includes a first graphicpattern 76 with twelve inner graphic patterns 56 drawn therein, as doesthe first graphic pattern 74 shown in FIG. 20. Herein, in the firstgraphic pattern 76, the colors of the inner graphic patterns 56 arrangedalong the four sides of the rectangular shape (square) are differentfrom one another for each side (directions and density of hatchingrepresent different colors in FIG. 21). In other embodiments, the marker75 shown in FIG. 21 may be used instead of the marker 51 shown in FIG.6. In such a case, the CPU 31 may calculate the direction of the marker75 (the direction with respect to the four directions (up, down, leftand right)) without using the second graphic pattern. That is, the CPU31 can identify the direction with respect to the four directions (up,down, left and right) of the marker 75 by detecting the color of theinner graphic patterns 56.

In the above embodiment, the marker 51 is provided with a white framearound the black first graphic pattern 55 for the purpose of making iteasier to detect the first graphic pattern 55. Therefore, the object tobe detected in the recognition process is the first graphic pattern 55inside the marker 51. In other embodiments, there may be no white frame,and the object to be detected in the recognition process may be theentire marker 51.

In the above embodiment, the marker 51 includes the second graphicpattern 57 at a position different from the first graphic pattern 55.The CPU 31 identifies the direction (the direction with respect to thefour directions (up, down, left and right)) of the first graphic pattern55 based on the direction in which the second graphic pattern 57 islocated with respect to the first graphic pattern 55 (S29). In otherembodiments, a marker including second graphic patterns drawn inside afirst graphic pattern may be used. For example, in the marker 51 shownin FIG. 6, the second graphic pattern 57 may be formed inside arectangular shape represented by the eight inner graphic patterns 56inside the first graphic pattern 55. Note that the second graphicpattern may be any graphic pattern as long as the direction thereof canbe identified, such as the arrow-shaped graphic pattern of the aboveembodiment. Then, the CPU 31 detects the second graphic pattern from thearea inside the rectangular shape formed by the four vertices, aftercalculating the four vertices in step S26. Then, the direction of thefirst graphic pattern with respect to the four directions (up, down,left and right) is identified based on the direction of the secondgraphic pattern detected. Also with this method, as with the aboveembodiment, it is possible to identify the direction of the firstgraphic pattern with respect to the four directions (up, down, left andright). Note however that when a marker with the second graphic patterndrawn inside the first graphic pattern is used, the inside graphicpatterns and the second graphic pattern are both included inside thefirst graphic pattern. Therefore, there is an increased possibility oferroneous detection when detecting the inside graphic patterns (ascompared with that of the above embodiment), and the precision of therecognition process may possibly decrease. In other words, since theabove embodiment uses a marker only with the inside graphic patternsinside the first graphic pattern, it is possible to more accuratelydetect the inside graphic patterns and to improve the precision of therecognition process.

While the marker 51 is a thin plate-like member with a predeterminedgraphic pattern drawn thereon in the above embodiment, the marker 51 maybe implemented by displaying a predetermined graphic pattern on adisplay apparatus in other embodiments. For example, in otherembodiments, another portable game apparatus with the marker 51displayed on its screen may be used as the marker 51.

(Variation Regarding Captured Image Used in Recognition Process)

In the above embodiment, the same captured image is used with thelow-precision area detection process (step S11), and the high-precisionmarker detection process (step S15). In other embodiments, images ofdifferent resolutions may be used in the two processes. Specifically,the area detection process may use an image with a resolution lower thanthe captured image used in the marker detection process. For example,where the game apparatus 1 produces a low-resolution image from an imagecaptured by the camera because, for example, such an image is needed ina process other than the recognition process, the low-resolution imagemay be used in the area detection process.

(Variation Regarding Precision of Recognition Process)

In the above embodiment, the process precision is varied between thearea detection process (step S11) and the marker detection process (stepS15) in the recognition process. The process precision is also variedbetween different processes (between step S21 and steps S23 and S25)within the marker detection process. In other embodiments, any precisionsettings may be used for the different processes in the recognitionprocess, and the processes may be performed with the same precision.

(Variation Regarding Calculation of Marker Position)

In the above embodiment, the CPU 31 calculates the center positions ofthe inner graphic patterns 56, and calculates the position of the marker51 (the positions of the four vertices P11 to P14) in the captured imagebased on the center positions. In other embodiments, the CPU 31 may usethe center positions of the inner graphic patterns 56 directly as theposition of the marker 51. For example, the CPU 31 may use the pointsP1, P3, P6 and P8 shown in FIGS. 9 and 17 as the position of the marker51. While positions of four points (the positions of the four verticesP11 to P14) are used as information representing the position anddirection of the marker 51 in the above embodiment, the position anddirection of the marker 51 can be identified when positions of three ormore points of the marker 51 are known. Therefore, in other embodiments,positions of three or five or more points may be used as informationrepresenting the position and direction of the marker 51. Therefore, theCPU 31 can detect the position and direction of the marker 51 based onthe center positions of three or more inner graphic patterns.

(Other Variations)

In the above embodiment, the game apparatus 1 executes the recognitionprocess on an image that is captured in real time by the external camera25 of the game apparatus 1. In other embodiments, the captured imageused in the recognition process may be an image that has been capturedin the past or a captured image that the game apparatus 1 obtains froman external device. While the external camera 25 is built in the gameapparatus 1 in the above embodiment, an external camera that can beattached to the game apparatus 1 may be used instead of the externalcamera 25 in other embodiments.

The above embodiment is directed to a case where a recognition processis used in an augmented reality technique. The object to be recognizedin the recognition process is a marker used in an augmented realitytechnique. The present invention is applicable to any image recognitionprocess of recognizing any image object from a captured image. Forexample, in other embodiments, the recognition process may be used in anapplication where the face of a user is recognized from a capturedimage, followed by a predetermined process performed on the recognizedface (e.g., adding thereon scribbles, etc.).

While the recognition process is performed by only one informationprocessing apparatus (the game apparatus 1) in the above embodiment, therecognition process may be allotted to a plurality of informationprocessing apparatuses in an image recognition system having a pluralityof information processing apparatuses that can communicate with eachother in other embodiments.

As described above, the present invention can be used as a game programor a game apparatus, for example, for the purpose of, for example,accurately detecting the position of the image object even if thecaptured image is blurred.

While the invention has been described in detail, the foregoingdescription is in all aspects illustrative and not restrictive. It isunderstood that numerous other modifications and variations can bedevised without departing from the scope of the invention.

1. A computer-readable storage medium having stored therein an image recognition program to be executed by a computer of an information processing apparatus for detecting a predetermined image object including a plurality of graphic patterns from a captured image captured by an image-capturing section, the image recognition program instructing the computer to function as: an image obtaining section for obtaining the captured image captured by the image-capturing section; a graphic pattern detection section for detecting the plurality of graphic patterns from the captured image; and an object detection section for detecting a position or direction of the predetermined image object by calculating center positions of the plurality of graphic patterns detected by the graphic pattern detection section.
 2. The storage medium according to claim 1, wherein: the predetermined image object includes a first graphic pattern with the plurality of graphic patterns drawn therein; the image recognition program instructs the computer to further function as an area detection section for detecting an area of the first graphic pattern from the captured image; and the graphic pattern detection section detects the plurality of graphic patterns from within the area detected by the area detection section.
 3. The storage medium according to claim 2, wherein: the graphic pattern detection section detects areas of the plurality of graphic patterns; and the object detection section includes a position calculation section for calculating center positions of the areas detected by the graphic pattern detection section.
 4. The storage medium according to claim 3, wherein: the graphic pattern detection section detects outlines representing the areas of the plurality of graphic patterns; and the position calculation section calculates center positions of the areas represented by the outlines detected by the graphic pattern detection section.
 5. The storage medium according to claim 4, wherein the position calculation section calculates the center position of each of the areas by obtaining positions of pixels representing the outline detected by the graphic pattern detection section and calculating an average value among the obtained positions.
 6. The storage medium according to claim 1, wherein: the object detection section includes a straight line calculation section for calculating a predetermined straight line based on the center positions of the plurality of graphic patterns; and the image recognition program instructs the computer to further function as a positional relationship calculation section for calculating a positional relationship between the predetermined image object and the image-capturing section based on the predetermined straight line calculated by the straight line calculation section.
 7. The storage medium according to claim 1, wherein each of the plurality of graphic patterns is circular.
 8. The storage medium according to claim 2, wherein: the area detection section detects an area of the first graphic pattern based on pixel values obtained at a first pitch across the captured image; and the object detection section calculates center positions of the graphic patterns based on pixel values obtained at a second pitch smaller than the first pitch across the area of the captured image detected by the area detection section.
 9. The storage medium according to claim 2, wherein: the image recognition program instructs the computer to further function as an outline extraction section for extracting an outline of the first graphic pattern in the area of the first graphic pattern detected by the area detection section; and the object detection section detects the plurality of graphic patterns from an area within the outline extracted by the outline extraction section.
 10. The storage medium according to claim 9, wherein: the outline extraction section extracts the outline of the first graphic pattern based on pixel values obtained at a predetermined pitch across the captured image; and the object detection section detects the plurality of graphic patterns from the area within the outline based on pixel values obtained at a pitch smaller than the predetermined pitch across the captured image.
 11. The storage medium according to claim 9, wherein: the image recognition program instructs the computer to function as a shape determination section for determining whether the outline extracted by the outline extraction section represents a shape of the first graphic pattern; and the object detection section detects the plurality of graphic patterns from within an area corresponding to the outline which is determined by the shape determination section to represent a shape of the first graphic pattern.
 12. The storage medium according to claim 2, wherein: the predetermined image object includes a second graphic pattern at a position different from the first graphic pattern; and the image recognition program instructs the computer to further function as a direction identification section for detecting the second graphic pattern from the captured image and identifying a direction of the predetermined image object based on a direction in which the second graphic pattern is arranged with respect to the first graphic pattern.
 13. The storage medium according to claim 3, wherein: the graphic pattern detection section detects the areas of the plurality of graphic patterns based on pixel values obtained at a predetermined pitch across the captured image; and the position calculation section calculates center positions of the areas detected by the graphic pattern detection section based on pixel values obtained at a pitch smaller than the predetermined pitch across the captured image.
 14. The storage medium according to claim 3, wherein: the object detection section further includes a number-of-areas determination section for determining whether the number of areas detected by the graphic pattern detection section is a number within a predetermined range; and the position calculation section calculates center positions of the areas detected by the graphic pattern detection section for the area of the first graphic pattern where a number of areas within the predetermined range have been detected.
 15. The storage medium according to claim 1, wherein: the predetermined image object includes three or more graphic patterns; and the object detection section detects a position and direction of the predetermined image object based on center positions of the three or more graphic patterns.
 16. The storage medium according to claim 1, wherein: a predetermined number, three or more, of graphic patterns are arranged on a predetermined straight line in the predetermined image object; and the object detection section calculates the predetermined straight line based at least on a number of center positions among all center positions calculated, the number being two or more and smaller than the predetermined number, so as to detect a position and direction of the predetermined image object based on the calculated predetermined straight line.
 17. An image recognition apparatus for detecting a predetermined image object including a plurality of graphic patterns from a captured image captured by an image-capturing section, the image recognition apparatus comprising: an image obtaining section for obtaining the captured image captured by the image-capturing section; a graphic pattern detection section for detecting the plurality of graphic patterns from the captured image; and an object detection section for detecting a position or direction of the predetermined image object by calculating center positions of the plurality of graphic patterns detected by the graphic pattern detection section.
 18. An image recognition system for detecting a predetermined image object including a plurality of graphic patterns from a captured image captured by an image-capturing section, the image recognition system comprising: an image obtaining section for obtaining the captured image captured by the image-capturing section; a graphic pattern detection section for detecting the plurality of graphic patterns from the captured image; and an object detection section for detecting a position or direction of the predetermined image object by calculating center positions of the plurality of graphic patterns detected by the graphic pattern detection section.
 19. An image recognition method for detecting a predetermined image object including a plurality of graphic patterns from a captured image captured by an image-capturing section, the image recognition method comprising: an image obtaining step of obtaining a captured image captured by the image-capturing section; a graphic pattern detection step of detecting the plurality of graphic patterns from the captured image; and an object detection step of detecting a position or direction of the predetermined image object by calculating center positions of the plurality of graphic patterns detected in the graphic pattern detection step.
 20. A marker used in an image display system, in which a captured image captured by an image-capturing section is subjected to a predetermined recognition process to calculate a position and direction of the marker within the captured image, and an image of a virtual object is produced based on the position and direction of the marker, so as to display an image obtained by synthesizing the image of the virtual object with the captured image, wherein: the marker comprises a plurality of graphic patterns whose center positions are calculated; and the center positions are used for calculating information representing the position and direction of the marker.
 21. A marker display apparatus for displaying a marker used in an image display system, in which a captured image captured by an image-capturing section is subjected to a predetermined recognition process to calculate a position and direction of the marker within the captured image, and an image of a virtual object is produced based on the position and direction of the marker, so as to display an image obtained by synthesizing the image of the virtual object with the captured image, wherein: the marker display apparatus displays on a screen a plurality of graphic patterns whose center positions are calculated; and the center positions are used for calculating information representing the position and direction of the marker. 